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[57] ABSTRACT 

A transcoding method and apparatus having an input former, 
a requantizer, a buffer, a prediction processor, and a com- 
posite switch. The transcoder has multiple feedback paths 
routed through the composite switch that are selected 
depending upon system requirements. The input former can 
further include an inverse quantizer, a summer, and a selec- 
tor. The selector connects different signal paths in the 
transcoder depending upon the coding mode of the input 
bitstream. The buffer can include a frame buffer, a QP buffer, 
and a comparator for generating a control signal. Further 
features of the transcoding method and apparatus include a 
frequency weighted thresholder and a computation resource 
manager. 

37 Claims, 11 Drawing Sheets 
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DEVICE AND METHOD FOR DIGITAL 
VIDEO TRANSCODING 

BACKGROUND OF THE INVENTION 

This invention relates to transcoders, and particularly to 
digital video transcoders for real time conversion between a 
first and a second coding scheme. 

Due to the fast advances in digital technology and VLSI 
(very large scale integration) technology and the acceptance 
of international standards, digital video now finds applica- 
tions in many areas. For example, multimedia, videoconfer- 
encing and videotelephony applications all utilize digital 
video. In each of these applications, occasions exist where 
data representing moving picture television or sound must 
be transmitted over long distances via a transmission link. 
Transmitting the data in digital form, however, requires high 
bandwidth communication channels and is expensive. To 
overcome these disadvantages, various techniques have 
been designed to compress the digitized video data and to 
reduce the bit rate required to transmit the coded video 
signals. 

Video compression techniques reduce the amount of data 
being transmitted by allowing for an acceptable degree of 
degradation in picture quality. Possible correlations that can 
be exploited to compress the digital data include the spatial 
correlations among neighboring pixels in an image and the 
temporal correlation between successive images. For 
instance, transform techniques reduce the amount of infor- 
mation needed to code a particular frame by removing the 
statistical redundancy among neighboring samples using 
prediction methods. One known form of data compression 
employs predictive coders where an original sample is 
predicted based upon past samples and a prediction error 
(the difference between the original and the predicted 
samples). Predictive coders can be used for speech, image, 
or video compression. 

The state-of-the-art for video compression is the hybrid 
coding method where predictive encoding is used in reduc- 
ing temporal redundancy and transform coding is used in 
ehrninating spatial correlation. The ITU-T, H.261 and H.263 
recommendations employ this hybrid coding technique to 
achieve data compression. The H.261 and H.263 guidelines 
suggest utilizing motion-compensated prediction for tempo- 
ral prediction, and discrete cosine transforms (DCT) for 
spatial domain processing. The coded data stream can be 
reconstituted into a series of video signals by a compatible 
decoder which decompresses the coded data stream. 

FIG. 1 illustrates a block diagram of a hybrid video 
encoder 10. According to the ITU-T recommendations, the 
video input and output consists of individual frames coded 
in either Common Intermediate Format (hereinafter "CIF') 
or Quarter Common Intermediate Format (hereinafter 
"QCIF"). Each frame of the video signal has one luminance 
component (Y) and two chrominance components (Cb and 
Cr). Typically, the chrominance components have only half 
of the resolution of the their luminance counterpart. For 
example, for a C1F video, the Y component has 288 lines of 
352 pixels while the Cb and Cr only have 144 lines of 176 
pixels. 

To encode a video frame, the frame is first segmented into 
non-overlapping square blocks. Each of the non-overlapping 
square blocks is called a macroblock (hereinafter "MB"), 
and the size of the macroblocks is NxN for luminance 
components and is (N/2)x(N/2) for chrominance 
components, where N is a predetermined integer. A typical 
value for N is 16. 
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The encoder of FIG. 1 shows that each MB can be coded 
either in the intra mode or in the inter mode. In the so-called 
intra mode, the input MB is directly fed to the transformer 
by a switch SI. Intra mode coding is selected to aid in 

5 combating transmission error propagation resulting from 
predictive systems or when there is no similar part in the 
previous video frame for prediction. Inter mode coding is 
normally selected to provide for data compression using 
predictive coding. 

1Q When the encoder codes the input video in the inter mode, 
the encoder searches for a NxN block of data in the 
previously decoded frame that minimizes a predetermined 
error function. This NxN block of data, called the prediction 
macroblock, is retrieved from a frame buffer by a motion 

15 compensator. The encoder then forms a prediction error 
macroblock by finding the difference between the input MB 
and the prediction MB at a summer. A transformer then 
transforms the prediction error macroblock output by the 
summer. The output of the transformer is then quantized by 

20 a quantizer. Typically, an entropy coder completes the 
encoding of the video data, and an output buffer smoothes 
out the variation in bit generation to achieve the desired 
bitrate for the output channel. An inverse quantizer is also 
coupled with the output of the quantizer. The inverse quan- 

25 tizer and an inverse transformer return the signal to its 
original format. This allows the outputs of the inverse 
transformer and the motion compensator to be summed and 
stored in a frame buffer for coding of the next frame. 
Further aspects of inter mode coding provide for skipped 

30 Macroblocks. Skipped, or non-coded, Macroblocks are 
formed when the prediction error is so small that after 
quantization, all the coefficients are quantized to zeroes and 
the motion vectors associated with the MB are zero. In 
H.261 format, the skipped MBs can be coded by differen- 

35 tially coding the macroblock address (MBA). For example, 
if an MBA of 3 is received for the current non-skipped MB 
is received, then it suggests that the previous 2 MB are 
skipped. In H.263, 1 bit is used to signal if the current MB 
is coded or not. If it is 1, then the current MB is skipped. If 

40 it's 0, then the current MB is non-skipped (or coded). 

FIG. 2 illustrates a block diagram schematic of a decoder 
20 known in the art. It may be viewed as a subset of the 
corresponding encoder shown in FIG. 1. An input buffer of 
the decoder buffers the incoming binary bit stream, and an 

45 entropy decoder decodes the coding mode, motion vectors 
and transform coefficients in the incoming bit stream. An 
inverse quantizer then inverse quantizes the transform 
coefficients, and an inverse transformer transforms the quan- 
tized coefficients. With the aid and direction of the decoded 

50 mode information and motion vector, a prediction loop 
generates an output video signal from the output of the 
inverse transformer. The prediction loop includes a summer 
that adds the output of the inverse transformer and the 
motion compensator to generate the output video signal, a 

55 frame buffer for storing and retrieving the output video 
signal, and a motion compensator for adjusting the signal 
from the frame buffer according to the motion vector. 

When an encoder-decoder pair are attached to the ends of 
the same communication channel, such as a network, the 

60 encode and decode system accurately transmits the digital 
data over the communication channel. However, in real 
applications, there are many situations where the two ter- 
minals are attached to different communication channels. 
Communication channels may have different operating rates 

65 or characteristics that complicate the encoding and decoding 
of the digital data. For example, FIG. 3 illustrate a commu- 
nication system having a first video terminal connected to 
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network 1 and operating at rate 1, and a second video that generates a corrected signal based upon an input bit- 
terminal connected to network 2 and operating at rate 2. A stream of digitized data and a feedback signal, and a 
bridge device, having a transcoder, must be used to allow requantizer that quantizes the corrected signal to form an 
terminal 1 to talk to terminal 2. The bridge device performs output signal having the desired quantization level. The 
both network protocol conversion and video transcoding, s requantizer also generates an error signal representative of 
Such a bridge device is commonly referred to as a multipoint the quantinization error in the output signal. The novel 
control unit (MCU) if more than two terminals are involved transcoder further includes a buffer for storing and retrieving 
in a conference. the quantinization error signal generated by the requantizer, 

Transcoders convert the incoming rate of the bit stream and a prediction processor for generating a prediction signal 

into a rate acceptable to the decoding or output end of the bit 10 based upon a delayed quantinization error signal received 

stream. Transcoders can be used to change the rate of a from the buffer. A composite switch generates and forwards 

digital video bit stream in a network, or to convert one video to the input former the feedback signal. The composite 

compression format to another. Additionally, transcoders switch generates the feedback signal by selecting between 

can convert a constant bit rate stream into a variable bit rate me delayed quantinization error signal and the prediction 

stream to save bandwidth through statistical multiplexing. 15 signal. Unlike prior art systems, this transcoder provides for 

Another important application of transcoding involves mul- selectable feedback paths, each path being capable of gen- 

tipoint videoconferencing where transcoders mix video sig- erating a feedback signal. Shorter and less computationally 

nals together for continuous presence teleconferencing. intensive feedback paths rapidly form substantially noiseless 

FIG. 4 illustrates a transcoder formed by cascading a feedback signals, while longer and more computationally 

video decoder with a video encoder. The decoder decodes 20 expensive feedback paths can be utilized when needed to 

the input bit stream into pixel data, and the encoder encodes process more complicated image signals, 

the intermediate video into the output bit stream with the One aspect of the invention provides for an input former 

desired algorithm and bitrate. This transcoder introduces having a selector that modifies the corrected signal based 

processing delay and requires significant implementation u PO° the coding mode of the input bitstream. 

complexity. 25 The selector connects different signal paths depending 

FIG. 5 is a block diagram of a more efficient video u pon whether the input bitstream is coded as an intra mode, 

transcoder as disclosed in: (1.) Eyuboglu, U.S. Pat. No. inter mode, or skipped block signal. 

5,537,440; (2.) M. Yong, Q.-F. Zhu, and V. Eyuboglu, "VBR Other features of the invention include a buffer having a 

transport of CBR encoded video over ATM networks," Sixth frame buffer, a QP buffer, and a comparator. The frame 

Int. Workshop on Packet Video, Sept. 1994; and (3.) D. G. 30 buffer stores and retrieves the quantinization error signal 

Morrison, M. E. Nilsson and M. Ghanbari, "Reduction of the generated by the requantizer. The QP buffer and the com- 

bit-rate of compressed video while in its coded form," Sixth parator provide additional control signals to the transcoder 

Int. Workshop on Packet Video, Sept. 1994. As shown in for the processing of data encoded in the skipped block 

FIG. 5, a modified version S, of the quantinization error mode. In particular, the comparator generates a control 

vector is subtracted from a received quantized vector Y f to signal based upon the current quantinization step size uti- 

form a difference vector E-. The difference vector E- is lized by the requantizer and a past step size stored in the QP 

requantized to obtain the transcoded vector Y/. A quantini- buffer. 

zation error vector calculator then computes the inverse [ n accordance with further aspects of the invention, the 

transformed quantinization error vector d £ equal to Af^D/], composite switch can include a first switch and a second 

where D/-Y/-E/ is the quantinization error vector and A, 1 switch. The first switch selectively determines an input 

is an inverse transformation. A modifying circuitry deter- signal to the composite switch, and the second switch 

mines the modified quantinization error vector S, based on selectively determines an output signal from the composite 

the past vectors d/. The modifying circuitry can include a switch. 

transformer, a motion compensator, and a frame buffer. The { of the ^venton mcludes a frequency 

frame buffer stores past values of the vectors d,\ weighted thresholder operably coupled with the input 

The approach illustrated in FIG. 5, however, does not former. The frequency weighted thresholder generates a 

fully account for the processing of skipped macroblocks. frequency weighted corrected signal by applying frequency 

This transcoder can allow noise introduced during the dependent threshold limits to the corrected signal. The 

transcoding of past frames to remain in the image signal of 5Q corrected signal formed generates a video image more suited 

a skipped macroblock. This effect creates low quality image t 0 the particulars of human vision. An additional feature of 

signals that represent the background in a video signal. the invention is a computation resource manager. The 

Additionally, the transcoder shown in FIG. 5 fails to provide resource manager monitors the computational usage status 

for a feedback path that does not require motion compen- 0 f the transcoder and controls the computational usage of the 

sat * 0D * 55 transcoder in response to the monitored usage. 

Accordingly, it is an object of the invention to provide for nP^rPTPTTOM OP thp np awinpq 

a more efficient and cost effective transcoder. BRIEF DES CRIPTION OF THE DRAWINGS 

A particular object of the invention is to effectively FIG - 1 k a block diagram schematic of a prior art hybrid 

provide for the processing of skipped macroblocks. video encoder; 

Another object of the invention is to provide a transcoder 60 FIG 2 is a block diagram schematic of a prior art hybrid 

having a plurality of feedback paths. Vlde0 decoder; 

TTiese and other objects of the invention are evident in the FIG ' 3 two prior art video networks connected 

following description. throu 8 h a bridging device having a transcoder; 

^tww.™, ^ Tvr ,_ FIG. 4 shows a block diagram of a known prior art 

SUMMARY OF THE INVENTION fi5 transcoder; 

The invention achieves the foregoing objects with a FIG. 5 shows a block diagram of another prior art 

transcoding method and apparatus having an input former transcoder; 
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FIG. 6 is a block diagram of a transcoder in accordance 
with the present invention; 

FIG. 7 illustrates a further embodiment of the transcoder 
shown in FIG. 6 having a detailed diagram of a requantizer 
and a prediction processor; 5 

FIG. 8 shows a block diagram of the transcoder of FIG. 
6 having a detailed diagram of an input former and of a 
predictor; 

FIG. 9 is a detailed block diagram of the transcoder of 
FIG. 6; 10 

FIG. 10 is a detailed block diagram of the transcoder of 
FIG. 6 having a frequency weighted thresholder; 

FIG. 11 is an example of a threshold matrix associated 
with the frequency weighted thresholder of FIG. 10; 15 

FIG. 12 shows the transcoder of FIG. 6 having a com- 
putation resource manager; and 

FIG. 13 illustrates a flow chart of the transcoding method 
in accordance with the present invention. 

DETAILED DESCRIPTION OF THE 20 
INVENTION 

FIG. 6 illustrates a transcoding system having a syntax 
decoder 30, a transcoder 40 according to the invention, and 
a syntax encoder 60. The transcoding system can be incor- 2 s 
porated into a communication system so that the transcoder 
40 acts as a bridge between two networks. The syntax 
decoder 30 receives an input bit stream 32 from one 
network, and generates an input signal 34 that is fed to the 
transcoder 40. When operating within the confines of the 3Q 
H.261 standards, the input signal 34 generated by the syntax 
decoder 30 is macroblock data and side information from the 
input bit stream 32 pertaining to framing, inter/intra mode, 
step size, and motion vectors. The syntax encoder 60 
receives an output signal 36 from the transcoder 40 and 35 
generates an output bit stream 62 in response thereto. 

The transcoder 40 includes an input former 42, a requan- 
tizer 44, a buffer 46, a prediction processor 48, and a 
composite switch 50. The input former 42 receives the input 
signal 34 from the syntax decoder 30 and a feedback signal 40 
52 from the composite switch 50. The input former 42 
generates a corrected signal 54 in response to the received 
input signal 34 and feedback signal 52. The requantizer 44 
quantizes the corrected signal 54 and generates a quantized 
output signal 36 and a quantinization error signal 56. The 45 
buffer receives the quantinization error signal 56 and gen- 
erates a delayed quantinization error signal 58. The delayed 
quantinization error signal 58 is routed both to the prediction 
processor 48 and to the composite switch 50. The composite 
switch 50 receives a prediction signal 59 from the prediction 50 
processor and a delayed quantinization error signal 58 from 
the buffer. The composite switch 50 generates the feedback 
signal 52 by selecting between the delayed quantinization 
error signal 58 and the prediction signal 59. 

The transcoder 40 shown in FIG. 6 advantageously 55 
includes two quantinization error feedback paths that lead 
from the requantizer 44 and proceed through the buffer 46 to 
the input former 42. The shorter feedback path proceeds 
directly from the buffer 46 to the composite switch 50. This 
shorter feedback path occurs when the composite switch 50 60 
creates a feedback path that leads from the buffer 46 directly 
to the input former 42. The longer feedback path proceeds 
from the buffer 46, through the prediction processor 48, to 
the composite switch 50. This longer feedback path is 
created when the composite switch 50 forms a feedback path 65 
that proceeds from the prediction processor 48 to the input 
former 42. 
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The shorter feedback path generates a feedback signal that 
is frequently more accurate than a feedback signal generated 
by the longer feedback path. The longer feedback path 
through the prediction processor 48 creates additional dis- 
tortions in the feedback signal 52 because of the additional 
processing of the delayed quantinization error signal 58 that 
occurs in the prediction processor 48. The prediction signal 
59 is frequently a less accurate representation of the quan- 
tinization noise than the delayed quantinization error signal 
58, and the prediction signal 59 typically takes longer to 
generate than the immediate routing action of the delayed 
quantinization error signal 58. However, the longer feedback 
path proves to be effective when spatial manipulation or 
spatial filtering of the feedback signal 52 is necessary. For 
instance, the longer feedback path proves particularly effec- 
tive when the signal must undergo motion compensation. 
Accordingly, the invention provides for a selectable feed- 
back path for generating the feedback signal 52. 

In operation, the transcoder 40 of FIG. 6 accepts a binary 
digital videostream generated by a remote encoder accord- 
ing to a predetermined syntax. The transcoder 40 also 
outputs a binary digital videostream having a bit rate that 
may differ from the rate of the original input bit stream 32. 
The output bit stream 62 can be formed having the same 
syntax as the original input bit stream 32 or it can be formed 
having a different syntax than the original input bit stream 
32. 

The syntax decoder 30 performs a partial decoding of the 
input bit stream 32 and thereby generates a set of signals, 
including: coding mode signals, motion vectors, and trans- 
form coefficients. The input former 42 receives the trans- 
form coefficients from the syntax decoder 30 and a feedback 
signal 52 from the composite switch 50. The input former 42 
generates, in response to the feedback signal 52 and to the 
transform coefficients, a corrected signal 54. The corrected 
signal 54 represents an input signal 34 that has been modi- 
fied by the feedback signal 52. Typically the feedback signal 
52 is a measure of the quantinization error introduced by the 
requantizer 44, and accordingly the corrected signal 54 is 
updated to counteract the noise introduced by the requan- 
tizer 44. 

The requantizer 44 quantizes the output of the input 
former 42, i.e. the corrected signal 54, to generate a quan- 
tized output signal 36 having a bit rate that differs from the 
bit rate of the input bit stream 32. The requantizer 44 also 
generates a quantinization error signal 56 based upon the 
difference between the input to the requantizer 44 and the 
output of the requantizer 44. Hie syntax encoder 60 receives 
the quantized output signal 36 from the requantizer 44 and 
encodes the output of the requantizer 44, together with the 
motion vectors, to generate the output bit stream 62 accord- 
ing to a predetermined syntax. Typically, the syntax is the 
same as that of the input bit stream 32. 

The buffer 46 receives the quantinization error signal 56 
from the requantizer 44. The quantinization error signal 56 
is a measure of the error introduced to the corrected signal 
54 during quantinization of the corrected signal 54. The 
buffer 46 generates a delayed quantinization error signal 58 
that represents a prediction of the quantinization noise for 
the next input signal 34. Generally, the delayed quantiniza- 
tion error signal 58 is based upon past values of the 
quantinization error signal 56. 

The prediction processor 48 receives the delayed quanti- 
nization error signal 58 from the buffer. The prediction 
processor 48 then generates a prediction signal 59 in 
response to the delayed quantinization error signal 58. The 
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prediction processor 48 modifies the delayed quantinizauon As shown in FIG. 7, the quantizer 78 is operably coupled 

error signal 58 in the spatial domain, while typically the to receive the corrected signal 54 from the input former 42. 

delayed quantinization error signal 58 is located in another The quantizer 78 quantizes the coefficients in the corrected 

domain, such as the frequency domain. The prediction signal 54 with step size Delta, where the step size is 

processor 48 operates within the spatial domain because 5 determined as a function of the difference between the bit 

certain aspects of the delayed quantinization error signal 58 rate 0 f ^ mput bit stream 32 and the desired bit rate of the 

may be more easily manipulated in the spatial domain. After output bil slream 62 . The step ^ further determined as 

the prediction processor 48 has modified the delayed quan- a ^ QCiion of the desired ^ ge quality . step size is 

tinization error signal 58 within the spatial domain the determined ^ M t0 achieve the desired ch e m bit rate 

prediction processor 48 transforms the modified delayed 10 whik ^ m or retainin ^ m Accordingly, the 

quantinization error signal 58 back to its origmal domain * a * ^ ^ ^ ^ ^ transcoder 

and outputs the signal as the prediction signal 59. AA r . . 

r . . r . f 40 of the present invention. 
The composite switch 50 receives the delayed quantini- 
zation error signal 58 from the buffer 46 and the prediction The current quantinization step is particularly important 
signal 59 from the prediction processor 48. The output of the 15 during the skipped block mode. Data regarding the current 
composite switch 50 is coupled with the input former 42 and quantinization step size aids in determining how to reduce 
is designated as a feedback signal 52. The composite switch noise in the output signal 36 during the skipped block mode 
50 is controllable to either create a signal path between the of operation. Accordingly, a step size signal 84, representing 
buffer 46 and the input former 42 (i.e. the shorter feedback the quantinization step size currently being utilized by the 
path), or to create a signal path between the prediction 20 quantizer is output by the quantizer. As shown in FIG. 7, the 
processor 48 and the input former 42 (i.e. the longer feed- buffer 46 receives the step size signal 84. 
back path). When the composite switch 50 selects the shorter 

feedback path the feedback signal 52 depends directly upon T^ requantizer 44 also includes an inverse quantizer 80 

the delayed quantinization error signal 58, and when the and a summer coupled with the quantizer 78 for generating 

composite switch 50 selects the longer feedback path the 25 a quantized error signal 56 representative of the noise added 

feedback signal 52 depends upon the prediction signal 59, to the corrected signal 54 by the quantinization process. In 

FIG. 7 illustrates further aspects of the transcoder 40 in particular, the inverse quantizer 80 is operably coupled to 

accordance with the invention. In particular, FIG. 7 shows an receive the output signal 36 from the quantizer 78. The 

expanded view of the syntax decoder 30, the syntax encoder inverse quantizer 80 performs the corresponding inverse 

60, the requantizer 44, and the prediction processor 48. 30 quantinization with the same step size. The summer is 

The syntax decoder 30 includes an input buffer 70 and an C0U P le f t0 u re <* ive the *>™&* signal 54 and is 

entropy decoder 72. The input buffer 70 stores and retrieves operably coupled with the inverse quantizer 80 to receive the 

the input bit stream 32 and the entropy decoder 72 decodes quantized signal 86. The summer generates a quantized error 

the input bit stream 32 into transform coefficients and side „ sl S na |^ ual t0 . the ^rence J)e^een me corrected signal 54 

information. TTie transform coefficients are forwarded to the 35 and the signal 86. The signal produced by the 

input former 42. The side information allows the syntax summer represents the quantimzation error generated during 

decoder 30 to correctly reconstruct the coded signal. Side me Quantinizauon process. 

information can include the source format (CIF or QCIF), The prediction processor 48 shown in FIG. 7 includes an 

quantizer 78 step size, inter/intra mode decision, and motion inverse transformer 88, a predictor 90, and a transformer 92. 

vectors. The encoder can also transmit side information to The inverse transformer 88 receives the delayed quantini- 

allow the transcoder 40 to skip a macroblock. For example, zation error signal 58 from the buffer 46 and generates a 

when all the coefficients in the block are zero and when there spatial signal 94. The inverse transformer 88 transforms the 

is little motion, the macroblock may be identified as a delayed quantinization error signal 58 to a domain more 

skipped macroblock in the side information. 45 appropriate for spatial modification. The predictor 90 

The syntax encoder 60 includes an entropy coder 74 and receives the spatial signal 94 and generates a spatial predic- 

an output buffer 76. The output signal 36 of the requantizer tion signal 96 that is output to the transformer 92. The 

44 is received by the entropy coder 74. The entropy coder 74 transformer 92 then operates upon the spatial prediction 

re-encodes the output signal from the requantizer 44 and signal 96 with a transform that is the inverse of the inverse 

forwards the re-encoded binary bits to the output buffer 76. 50 transformer 92. That is, the transformer 92 returns the signal 

The output buffer 76 stores the binary bit stream for trans- back to the domain of the delayed quantinization error signal 

mission along a communication channel in the form of an 58. 

output bit stream 62. The transformer 92 mcd m lne transcoder 40 is preferably 
The transcoder 40 of FIG. 7 also includes a detailed block a discrete cosine transform (DCT) and the inverse trans- 
diagram of the requantizer 44. The requantizer 44 includes 55 former 92 is preferably an inverse discrete cosine transform, 
a quantizer 78, an inverse quantizer 80, and a summer 82. The inverse discrete cosine transform is applied indepen- 
The quantizer 78 receives a corrected signal 54 and gener- dently to each 8x8 block in the macroblock to obtain the 
ates the output signal 36 in response thereto. Quantizers are transform coefficients as given by: 
known in the art and are used to create a discrete level 
representation of an input signal in accordance with a 60 spatial signal 94 (x, y) - 
quantinization step size. Typically, the quantinization step 

size ^can be adjusted with a control signal. As the magnitude JL I I c»C(v)F(* v)cosW2x + l>/i61cos[<2^i)v/i6] 

of the quantinization step size increases, the fineness of the 4 «»o v«o 
quantinization decreases, and vice-versa. It is further worth 

noting that quantizers suffer from error. This error, called 65 

quantinization error, is the difference between the input u,v,x,y=0,l,2, . . . , 7; 

value and the reconstruction value of the input signal. where x,y=spatial coordinates in the pel domain; 



3/15/05, EAST Version: 2.0.1.4 



5,870,146 

9 10 

u,v=coordinates in the transformed domain; composite switch 50 to generate a feedback signal 52. When 

the transcoder 40 is operating in skipped block mode, the 
c(u) ^ 1 . for u m 0 control signal directs the composite switch 50 to generate a 

5 During skipped block mode, the comparator 110 corn- 
er a o er cases; an current step s \ z& signal 84 with the delayed step 

c(y) - 1 /or v-o size signal from the QP buffer 108. If the current step size 

NT is less than the delayed step size, then the control signal 

l ^or all other cases; directs the composite switch 50 to generate a skipped block 

signal based upon the delayed quantization error signal 58. 

F(n v) - delayed quantization error signal. 10 j£ however, the current step size is more than the delayed 

The predictor 90 is preferably a motion compensator. The size > to te control ^ rect f f c composite switch 

motion compensator adjusts the spatial signal 94 to account 50 J? S enerate * skipped block signal having a null value, 
for movement in the image being transmitted. The motion A The transcoder 40 shown in HG. 8 also includes a 
compensator relies upon a motion vector that is extracted „ deIaile <; * a S ran ? of the .coi^te switch 50. The am^tc 
from the input bitstream by the entropy decoder 72 to 15 switchSOhasa i first switch 116 and a second switch 118 The 
implement motion compensation. Topically a motion vector fi u ret switch 116 cont u ro J? the selection of an input signal into 
is generated, in a remote encoder, by an algorithm that * c composite switch 50 and the second switch 118 controls 
searches the pixels in a previously reconstructed image to l c h A e f le . cll0n o{ ™ output signal from the composite switch 
find a window for which the "distance" between the pixels 50. An mtermediate signal path 120 connects the first switch 
in that window and the corresponding pixels in the current 20 116 and the 8000116 switch m 

block is a minimum. The spatial offset between the window 1110 first switch 116 sclccts ^ m P ut Sl ^ 1 to the swltch 
and the current block is defined as the motion vector. The from either the delayed quantization error signal 58 or the 
motion compensator then uses the motion vector to offset the prediction signal 59. Thus, the intermediate signal is based 
current image signal to obtain a more accurate data repre- upon either the delayed quantization error signal 58 or the 
sentation of the image. 25 prediction signal 59, depending upon the input terminal 

FIG. 8 illustrates further details of the transcoder 40 chosen. The second switch 118 selects an output signal from 
shown in FIG. 6. The illustrated input former 42 includes an either the feedback signal 52, the skip block signal 114, or 
inverse quantizer 100, a summer 102 and a selector 104. In a null terminal 122. The intermediate signal is routed to 
addition, further details of the buffer 46 as shown include a either the feedback signal 52, the skip block signal 114, or 
frame buffer 106, a QP buffer 108 and a comparator 110. 30 the null position, depending upon the output terminal cho- 

The buffer 46 shown in FIG. 8 receives a quantinization sen. 
error signal 56 and a step size signal 84 from the requantizer The switch control signal 112 from the buffer 46 control 
44 and generates a delayed quantinization error signal 58 the selection of the output terminal in the second switch 118. 
and a switch control signal 112 as output. The quantinization If the transcoder 40 is not operating in skipped block mode, 
error signal 56 is routed to the frame buffer 106. The frame 35 then the intermediate signal is connected with the feedback 
buffer 106 generates the delayed quantinization error signal signal 52. If the transcoder 40 is operating in skipped block 
58 based upon past values of the quantinization error signal mode, then the intermediate signal is connected with either 
56. For instance, the delayed quantinization error signal 58 the skip block signal 114 or the null terminal 122. 
generated can be the quantinization error signal 56 during Accordingly, when operating in skip block mode, the inter- 
previous frames. In accordance with this embodiment, the 40 mediate signal can be connected to the skip block signal, 
coefficient frame buffer 106 can simply be a memory ele- thereby forming a corrected signal 54 substantially equiva- 
ment for storing and retrieving the quantinization error lent to the intermediate signal. Alternatively, when ope rating 
signal 56. in skip block mode, the intermediate signal can be connected 

The buffer 46 also includes a QP buffer 108 for receiving with the null terminal 122, thereby forming a corrected 
the step size signal 84 from the requantizer 44. The QP 45 signal 54 having a null value. 

buffer 108 is typically a memory element large enough for FIG. 8 also details elements forming the input former 42. 

storing and retrieving the step size signal 84 for a video The input former 42 includes an inverse quantizer 100, a 
frame. The memory size of the QP buffer 108 is determined summer 102 and a selector 104. The inverse quantizer 100 
by the number of macroblocks in a videoframe multiplied by receives an input signal 34 from the syntax decoder 30 and 
the space required to store the step size signal 84. For a 50 generates a quantized input signal 124. The summer receives 
typical video coding method like H.261 and H.263, five bits the quantized input signal 124 from the inverse quantizer 
is enough to store the step size signal 84 because the step 100 and a feedback signal 52 from the composite switch 50. 
size signal 84 only takes integer values from one to 31. For The summer 102 generates an update signal 126 equal to the 
a QCIF video with frame size of 144x176, we have 99 sum of the input signals it receives. The selector 104 
macroblocks. Therefore, the minimum QP buffer 108 size 55 receives the quantized input signal 124, the update signal 
for this case is 495 bits. However, it should be noted, that in 126, and the skip block signal 114 and selects one of the 
many practical applications one byte (8 bits) is used to store three received signals to be the corrected signal 54. 
the step size signal 84 in order to save implementation The selector 104 provides different paths for generating 

complexity. the corrected signal 54 to account for various modes of 

The buffer 46 also can include a comparator 110 for 60 coding the image in the input bit stream 32. The coding 
comparing the step size signal with the output of the QP modes include intra mode, inter mode, and skip block mode, 
buffer 108, i.e. a delayed step size signal. The comparator For instance, in intra mode coding the input bit stream is 
110 generates a switch control signal 112 in response to the based on a single video image, rather than a comparison 
inputs received from the QP buffer 108 and the requantizer between frames of the input picture. In this case, the selector 
44 and in response to the current operating mode of the 65 104 forms the corrected signal 54 based solely upon the 
transcoder 40. When the transcoder 40 is not operating in quantized input signal 124. When the original macroblock is 
skipped block mode, the control signal 112 directs the coded in inter mode, the input bit stream is based upon a 
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comparison between video frames. Under these tortion added to the signal by the prediction processor 48 is 

circumstances, the selector 104 must couple the corrected eliminated. In particular, the delayed quantinization error 

signal 54 with the feedback signal 52. Particularly, in inter signal 58 contains less additive noise than the prediction 

mode, the corrected signal 54 is generated by routing the signal 59 and accordingly the delayed quantinization error 

sum of the quantized input signal 124 and the feedback 5 signal 58 produces a higher quality picture, 

signal 52 through the selector 104. If the original macrob- mG - 10 Pirates the transcoder 40 having a frequency 

lock coded in the input bit stream 32 is coded as a skipped weighted thresholder (FWT) 136. The FWT136 is operably 

macroblock, then the selector 104 bases the corrected signal coupled to the input former 42, particularly the FWT 136 

54 upon the skip block signal 114 received from the com- receives the corrected signal 54 from the input former 42 and 

posite switch 50. 10 generates a signal that is received by the quantizer 78 in the 

FIG. 9 is a detailed schematic of the transcoder 40 and requantizer 44. The FWT 136 improves the picture quality 

various control signals. A S-control signal 130 and a of the out P ut si g° al 36 generated by the requantizer 44. 

M-control signal 132 are generated when the entropy 11 » wcl1 that human vision systems can tolerate 

decoder 72 decodes the side information from the input bit ni S her distortions for higher frequency components than for 

stream 32. A Q-control signal 134 is generated by the output 15 Iow frequency components. However, in most video coding 

buffer 76 and is input to the quantizer 78. ' systems, such as H.261 and H.263, all the transform coef- 

The Q-control signal 134 is used to control the quantini- ficients are treated equally. The FWT 136 takes advantage of 

zation step size of the quantizer 78. The Q-control signal 134 the particulars of human vision to improve picture quality by 

varies as a function of the fullness of the output buffer 76 applying different weights to the various frequency coeffi- 

memory. Accordingly, the quantizer 78 is forced to be 20 cient * of the video image. 

responsive to the fullness of the output buffer 76. For The FWT 136 is a filter that is frequency-dependent. The 

instance, the quantinization step size for a group of data can F ^ rr 136 deludes thresholds assigned to various frequency 

be determined based upon the quantity of codes stored in the components across the spectrum. The FWT 136 compares 

output buffer 76 at the time after the completion of the me input transform coefficients to the thresholds at a given 

coding of the preceding group of data. The Q-control signal 25 frequency component. If the input transforms coefficients do 

134 prevents buffer overflows and underflows by decreasing no1 exceed the threshold at the given frequency component, 

the quantinization step size. Decreasing the quantinization ihGQ the transform coefficient is set to zero. If the incoming 

step size increases the fineness of the quantinization thereby transform coefficient exceeds the threshold value at the 

increasing the amount of data in the output signal 36. g iven frequency, then the transform coefficient is left 

Similarly, increasing the quantinization step size decreases 30 unmodified. The threshold values differ across the range of 

the fineness of the quantinization thereby decreasing the frequency being viewed by the FWT 136. In particular, the 

amount of data in the output signal 36. thresholds are higher for higher frequency components than 

The S-control signal 130 is produced at the entropy the y ^e for low frequency components, 

decoder 72 and is used to control the selector 104. In FIG « 11 illustrates an example of a threshold matrix 138 

particular, the S-control signal 130 determines the routing 35 forming the FWT 136 shown in FIG. 10. The matrix shown 

path to form the corrected signal 54. If the signal decoded at m FIG. 11 is for use with a DCT transform having an 8x8 

the decoder indicates that the macroblock is coded in intra block size - ^ real threshold of the FWT 136 are derived as 

mode, then the S-control signal 130 causes the selector 104 follows: 
to route quantized input signal 124 through the selector 104. 

If the macroblock is coded as inter mode, then the S-control 40 T ir w ti 

signal 130 causes the selector 104 to route the update signal where i=0,l, ... ,7; and 

126 through the selector 104. In addition, if the original j=0,l, • • ,7, 

macroblock is coded as a skip block, then the S-control In the above equation i is the row index and j is the 

signal 130 causes the selector 104 to route the skip block column index in the DCT transform domain, T« is the 

signal 114 through the selector 104. 45 threshold to be used for coefficients at location (i j), W, y is 

FIG. 9 also shows an M-control signal 132 that is gen- the corresponding weighting factor shown in the matrix of 

erated at the decoder stage from the input bit stream 32. The FIG. 11, and QP is half of the quantinization step size used 

M-control signal 132 directs the operation of the composite for quantizing the current macroblock for data, 

switch 50 and the motion compensator depending upon the FIG. 12 illustrates a further aspect of the invention for 

encoding mode of the video image in the input bit stream 32. 50 increasing the efficiency of the transcoder 40. The transcoder 

If the decoded input bit stream 32 further indicates that only 40 shown includes a computation resource manager 140 

interframe coding is used without motion compensation, operably coupled with the syntax decoder 30 and with the 

then the M-control signal 132 directs the composite switch input former 42 for monitoring the system load and for 

50 to connect the path between the delayed quantinization controlling the transcoder 40 in response to the system load, 

error signal 58 and the feedback signal 52. In comparison, if 55 The computation resource manager 140 can also be coupled 

the decoded input bit stream 32 indicates that motion with the composite switch 50 and the prediction processor 

compensation is used, then the M-control signal 132 directs 48 for controlling computational usage by the transcoder 40. 

the composite switch 50 to connect the path between the The computation resource manager is coupled to and directs 

prediction signal 59 and the feedback signal 52. the operations of the input former, the composite switch, the 

For relatively low bit rate digital video applications such 60 prediction processor, and the buffer using computational 

as videoconferencing and videotelephony, the majority of control signals. The computation resource manager proves 

the macroblocks will be coded without motion compensa- particularly effective in systems, such as MCUs, where 

tion. For these macroblocks, the present invention directly computation resources are precious, 

fetches their delayed quantinization error signal 58 from the The computation resource manager 140 continuously 

buffer 46 without going through the prediction processor 48. 65 monitors the system load by monitoring the input buffer 70 

The advantage of this is two-fold: (1) implementation com- fullness. When the input buffer 70 fullness is high, the 

plexity is significantly reduced, and (2) the additional dis- system typically cannot process the input data fast enough 
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and accordingly the load on the system must be reduced. The transcodes an input signal coded in either intra, inter, or skip 

computation resource manager 140 can modify the load on block mode. The illustrated method also efficiently 

the system through various means. The computation transcodes the input signal by selectively providing for 

resource manager 140 can alter the computation resource motion compensation of the input signal, 
usage by directing the actions of either the input former 42, 5 The flow chart starts at ste P 150 - After ste P 150 » logical 

the prediction processor 48, or the composite switch 50. flow proceeds to step 152. At step 152, the input bitstream 

For instance, the computation resource manager 140 can B decoded. After decoding, the transcoding system identifies 

control the composite switch 50 to stop the processing of the coding mode of the input bitstream at step 154 IT* 

skipped macroblocks. Additionally, the computation coding mode can be selected from intra, inter, and skip block 

resource manager 140 can instruct the input former 42 to 10 m0 A f S «° .°? .u . a u .u n 

. t f » , f t . » t t At decision box 156, the system identifies whether all 

disable the feedback aspects of the system to reduce the ski d Wocks afe ' leled f f ski ^ blocks need t0 

computational usage of the transcoder 40 Particularly, the bfi ^ then mQU6i oceeds to decision box 

computation resource manager 140 can force the selector 158 olherwise logical control proC eeds to decision box 164. 

104 in the mput former 42 to form a path between the At decision ^jx 158> me transcoding system determines 

quantized input signal 124 and the corrected signal 54. This 15 whether the current quantization step size is less than the 

position of the selector 104 reduces the computational previous quantization step size. If the current step size is 

requirements of the system because it cuts off the feedback less, then control proceeds to action box 160. If the current 

path generated through the feedback signal 52 and the step size is greater, then control returns to decision box 156. 

skipped block signal. Moreover, the computation resource At action box 160, the transcoding system obtains an error 

manager 140 can instruct the prediction processor 48 to 20 macroblock from the buffer 46. This error macroblock is 

disable all motion compensation in order to reduce the then applied against the current image signal. After box 160, 

computational usage of the transcoder 40. control proceeds consecutively to actions box 162. At box 

The computation resource manager 140 can be imple- 162, the transcoding system encodes the output signal 36 

mented with a state variable diagram, where the state usin g entropy coder 74. Afterwards, logical flow of the 

variable monitors and stores the sum of data indicative of the 25 transcoding system returns to decision box 156. 

system load. When the state variable rises above a particular At decision box 164, the transcoding system determines 

threshold, the computation resource manager 140 acts to whether the coding mode of the input bitstream is intra 

reduce the system threshold. The computation resource coding. If the bitstream is not intra coded, i.e. the bitstream 

manager 140 can continue taking further actions as the state is inter coded, then control proceeds to action box 166. If the 

variable rises above additional thresholds. In addition, the 30 bitstream is intra coding, the control proceeds to action box 

computation resource manager 140 can remove those 165 where the coefficients are decoded. After box 165, 

aspects of the system which reduce the computational load logical flow proceeds to action box 168. 

as the state variable falls. For instance, when the state At decision box 174, the transcoding system determines 

variable is high, the input former 42 may only generate the whether the input bitstream requires motion compensation, 

corrected signal 54 based upon the quantized input signal 35 If motion compensation is required, then control proceeds to 

124 and the motion compensator processor may be deacti- action box 176, where motion compensation is performed by 

vated. In comparison, when the state variable falls, the the prediction processor 48. If motion compensation is not 

computation resource manager 140 may choose to required, then control proceeds to box 178. At box 178, the 

re-activate the prediction processor 48. feedback signal 52 is selectively formed based on either the 

In another aspect of the invention, the computation 40 delayed quantization error signal 58 or the prediction 

resource manager 140 can modify the system load by si S nal 5 ^ Logical flow then proceeds to action box 166. 

changing the coding mode of the input bitstream 32. The At box 166, the input former 42 forms a corrected signal 

computation resource manager generates an intra refresh . 54 based upon the input bitstream 32 and a feedback signal 

request signal 141 that is coupled with a remote encoder. The 52 - Afterwards, at action box 168, the transcoding system 

remote encoder controls the coding mode of the input 45 quantizes the corrected signal to form an output signal 36. 

bitstream and the intra refresh request signal 141 directs the After box 168, logical flow proceeds to action box 170. 

operation of the remote encoder. Particularly, the control At action box 170, the system generates an error signal 

signal 141 can direct the remote encoder to encode the input representing the quantization error of the quantizer used in 

bitstream 32 in either intra mode or inter mode coding. ste P 168. After box 170, the system buffers the error signal 

Accordingly, to reduce the system load the computation 50 at step 172. After box 172, flow proceeds to box 180. 

resource manager can force the remote encoder to encode At box 180, the entropy coder 74 performs entropy coding 

the input bitstream in intra mode. pn the output signal 36. At box 182, the transcoding system 

When the computation resource manager forces the proceeds to the next macroblock of data that requires 

transcoder to operate in a computationally less intensive processing. 

status, the produced picture quality may deteriorate. The 55 The invention being thus disclosed and described in 

deterioration can be stopped by intra coding updating at the connection with the illustrated embodiments, variations and 

remote encoder. However, excessive use of intra -coding can modifications thereof will occur to those skilled in the art, 

also result in lower picture quality. Hence, the intra mode a °d are intended to be included within the scope of the 

request signal is only sent by the computation resource invention, as defined by the claims appended hereto, 

manager when the noise level exceeds a predetermined 60 Wnat is claimed is: 

threshold and when the load on the computational usage of 1- A transcoder for transcoding an applied digitally 

the system has exceeded another predetermined level. The encoded input signal and generating an output signal, com- 

noise level can be gauged by providing a counter that counts prising: 

the number of macroblocks whose coding mode is forced to an input former for generating a corrected signal from 

change during transcoding. 65 said input signal and a feedback signal, 

FIG. 13 illustrates a flow chart for implementing the a requantizer operably coupled to said input former for 

methods steps of the current invention. The method shown quantizing said corrected signal based upon a prede- 
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terminal quantization step size, and requantizer gen- a comparator operably coupled with said OP buffer, and 

erating said output signal and a quantization error said requantizer said comparator generating a switch 

signal, control signal based upon a comparison between said 

a buffer operably coupled to said requantizer for receiving , , ste ? *** ^ and a buffered step size signal, 

said quantization error signal and for generating a 5 " A transcoder according to claim 1, wherein said 

delayed quantization error signal, composite switch further composes 

. , , „ a first switch operably coupled with said buffer and said 

a prediction processor operably coupled with said buffer prediction processor, said first switch forming an inter- 

for generating a prediction signal as a function of said mediate signal by selecting between said delayed quan- 

delayed quantization error signal, and ^ tinization error signal and said prediction signal, and 

a composite switch operably coupled with said input a second switch operably coupled with said first switch 

former and either of said buffer and said prediction for selectively connecting said intermediate signal 

processor, said composite switch forming said feed- between a plurality of output paths, 

back signal by selecting between said delayed quanti- 12. A transcoder according to claim 1, further comprising 

zation error signal and said prediction signal, a frequency weighted thresholder operably coupled with 

wherein said prediction processor includes * said "V* ^mer said thresholder generating a frequency 

an inverse transformer operably coupled to receive and we if te A d corrected signal. 

m 0 „ •„,„/,• ft 'i„ »;«„c.f rt ™ rt „ -j 13. A transcoder according to claim 12, wherein said 

pertorm an inverse linearly transiorm on said c • . . . . . ^? - ' , 

\ . , s 4 . . J t frequency weighted thresholder applies frequency depen- 

delayed quantization error signal, , ; t , J , . # v . • Y 

4 n , , rr. . , . dent threshold limits to said corrected signal, 

a predictor operably coupled with said inverse trans- 2 q *a a , a a- . t • * r\u 

p c : . r . . c ... zu 14. A transcoder according to claim 1, further comprising 

former for receiving a signal from said inverse c ? • . r 

, a *- i A *- a resource manager for monitoring the computational usage 

transformer and generating a spatial prediction . . c * , , c & 4 ... * , , 6 

si nal and status of said transcoder and for controlling said transcoder 

« r ' i « •« j. p in response thereto, 

a transformer operably coupled with sad predictor for * . ^ , « u 

linearly transforming said spatial prediction signal. 25 15 ' f transcoder according to clam, 14, wherein said 

l.Atransc^eraccordingtoclaiml.whereinsaidrequan- JC! reS0UrCe mana8e ' " ^ 

tizer furmer comprises. 16. A transcoder according to claim 14, wherein said 

a quantizer operably coupled to receive said corrected com p Uta tion resource manager controls said prediction pro- 
signal from said input former, said quantizer generating cessor to modify the computational usage of said transcoder. 
said output signal at a new bit rate, 30 17. A transcoder according to claim 14, wherein said 

an inverse quantizer operably coupled with said quantizer computation resource manager controls said prediction pro- 

for generating a quantized signal, and cessor to modify the computational usage of said transcoder. 

an adder operably coupled to receive said quantized signal 18. A transcoder according to claim 14, wherein said 

and said corrected signal, said adder generating said computation resource manager controls said composite 

quantization error signal representative of error intro- 35 switch to modify the computational usage of said transcoder. 

duced by said requantizer. 19. A transcoder according to claim 14, wherein said 

3. A transcoder according to claim 2, wherein said quan- computation resource manager is operably coupled with a 
tizer generates a step size signal identifying the current remote encoder and wherein said computation resource 
quan tinization step size. manager controls the coding mode of said remote encoder. 

4. A transcoder according to claim 1, wherein said inverse 40 20. A transcoder according to claim 19 further comprising 
transformer is an inverse discrete cosine transformer and a counter, wherein said computation resource manager gen- 
wherein said transformer is a discrete cosine transformer. erates an intra refresh request signal for controlling said 

5. A transcoder according to claim 1, wherein said pre- remote encoder when said counter exceeds a predetermined 
dictor is a motion compensator. threshold. 

6. A transcoder according to claim 1, wherein said input 45 21. A transcoder according to claim 20, wherein said 
former comprises: counter counts the number of macroblocks affected by 

an inverse quantizer for forming a quantized input signal, computational overload. 

anc j " w 22. A method for transcoding an applied digitally encoded 

a summer for generating an update signal representative 50 generating an output signal, said method 

of the sum of said feedback signal Ind said quantized com P nsm 8 ^ successive steps of: 

input signal. generating a corrected signal as a function of said input 

7. A transcoder according to claim 6, wherein said input si S nal and a feedbac k signal, 

former further comprises a selector for generating said quantizing said corrected signal based upon a predeter- 

corrected signal by selecting between said update signal and 55 mined quantization step size to from said output signal, 

said quantized input signal. determining a quantization error signal based upon said 

8. A transcoder according to claim 7, wherein said input output signal and said corrected signal, 

former further comprises a selector for generating said buffering said quantization error signal and generating a 

corrected signal by selecting between said update signal, delayed quantization error signal, 

said quantized input signal, and a skip block signal. 60 forming a prediction signal as a function of said delayed 

9. A transcoder according to claim 1, wherein said buffer quantization error signal, and 

includes a frame buffer for storing said quantinization error determining a feedback signal by selecting between said 

s *g na l- delayed quantization error signal and said prediction 

10. A transcoder according to claim 9, wherein said buffer signal, 

further comprises 65 wherein said step of forming a prediction signal includes 

a QP buffer operably coupled for receiving and buffering generating a spatial signal by inverse linearly trans- 

a step size signal, and forming said delayed quantization error signal, 
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generating a spatial prediction signal by compensating 30. A method according to claim 22, further comprising 

said transformed signal with a motion vector, and the steps of monitoring and controlling a computational 

generating said prediction signal by linearly transform- usage status, 

ing said spatially corrected signal 31. A method according to claim 30, further comprising 

23. A method according to claim 22, further comprising 5 ^ st Q £. 
the step of identifying the coding mode of said input signal. 

24. A method according to claim 23, wherein said step of incrementing a state variable as said computational usage 
generating said corrected signal further comprises selec- status rises above a predetermined level, and 

lively forming said corrected signal depending upon said reducing said state variable as said computational usage 

identified coding mode. slatus fal]s be low said predetermined level. 

the 2 fte A s^f th0d aCC ° rdiDg l ° ° laim 24 ' fariher com P risin S 32. A method according to claim 31, wherein said step of 

e steps o . . , . . .j o j, t controlling said computational usage status further com- 

c • . « » i t c -j , 4 signal as a function of said state variable. 

forming said corrected signal as a function of said update „ * A , - ,. . . ... . . c 

signal when said identified coding mode is inter mode. 35 33 - A method according to claim 31, wherein said step of 

26. A method according to claim 24, further comprising controlling said computational usage status further com- 
the step of forming said corrected signal as a function of a prises selecting a signal path for generating said feedback 
quantized input signal when said identified coding mode is signal as a function of said state variable. 

intra mode. 34. A method according to claim 31, wherein said step of 

27. A method according to claim 26, further comprising 20 controlling said computational usage status further com- 
the steps of: prises generating a computational control signal as a func- 

forming a skip block signal, and tion of said state variable, said computational control signal 

determining said corrected signal as a function of a skip controlling at least one element selected from the group 

block signal when said identified coding mode is consisting of an input former, a composite switch, a buffer, 

skipped block mode. 25 an d a prediction processor. 

28. A method according to claim 27, wherein said step of 35. A method according to claim 30, wherein said con- 
generating said skip block signal further comprises the steps trolling step further includes controlling the coding mode of 
°£ a remote encoder. 

generating a step size signal identifying the current quan- 36. A method according to claim 35, further including the 
tinization step size, 30 step 0 f generating an intra refresh request signal for con- 
forming a switch control signal as a function of said trolling said remote encoder when said counter exceeds a 
generated step size signal, said switch control signal predetermined threshold. 

directing the formation of said skip block signal. 37. A method according to claim 36, further including the 

29. A method according to claim 22, wherein said step of step of counting the number of macroblocks affected by 
quantizing said corrected signal further includes weighting 35 computational overload. 

said corrected signal with frequency dependent threshold 

limits. ***** 
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[57] ABSTRACT 

A method and apparatus for compressing motion pic- 
tures is disclosed. The frames of the motion picture are 
divided into adjacent groups. Each group is treated as a 
three-dimensional image. The three-dimensional image 
is then filtered via a three-dimensional FIR filter to 
generate three-dimensional component images that are 
more efficiently quantized. The degree of quantization 
of each component image is determined in part by the 
spatial frequencies represented by the component image 
in question. For motion pictures derrived from inter- 
laced scanning devices, the quantization of specific 
component images is altered to prevent artifacts. 

17 Claims, 8 Drawing Sheets 
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METHOD AND APPARATUS FOR CODING 
MOTION PICTURES 

BACKGROUND OF THE INVENTION 5 

The present invention relates to methods and appara- 
tuses for reducing the amount of data needed to repre- 
sent an image, and more particularly, to a novel method 
and apparatus for compressing motion pictures. 

Images are conventionally represented by a two-di- 10 
mensional array of values in which each value repre- 
sents a property of the image at a corresponding point 
on the image. In the case of gray-scale images, a single 
number representing the gradations of intensity from 
white to black, referred to as the gray scale, is stored. In 15 
the case of color images, each "value" is a vector whose 
components represent the gradations in intensity of the 
various primary colors, or some alternative color code, 
at the corresponding point in the image. 

A motion picture comprises a sequence of such im- 20 
ages. Typically, thirty or more images are needed for 
each second of viewing time. 

This representation of a motion picture corresponds 
to the output of a typical image-sensing device such as 
a television camera. Such a representation is convenient 25 
in that it is easily regenerated on a display device such 
as a CRT tube. However, the number of bits needed to 
represent the data is prohibitively large for many appli- 
cations. A single 512x512 gray-scale image with 256 
gray levels requires in excess of 256,000 bytes. At 30 30 
frames per second, a communication channel with a 
bandwidth of approximately 64 million bits per second 
is needed to transmit the motion picture. A full color 24 
bit per pixel motion picture would require a bandwidth 
of more than 190 million bits per second. 35 

This bandwidth is significantly greater than that 
available for many communications purposes. For ex- 
ample, high quality consumer telephone lines are typi- 
cally limited to 64 thousand bits per second. Hence, 
commercially viable picture telephone systems require 40 
some form of image compression. 

Image compression systems used in motion picture 
compression applications make use of the redundancy 
between frames to reduce the amount of information 
that must be transmitted over the bandwidth-limited 45 
communication link. Many scenes in a motion picture 
include portions that are constant from frame to frame 
for several seconds, if not minutes. Prior art compres- 
sion schemes attempt to divide the scene into the con- 
stant portion and the portion that changes from frame to 50 
frame. 

The constant portion need be sent only once. Hence, 
if the constant portion comprises a significant fraction 
of the scene, considerable image compression can be 
realized. For example, if the entire scene were constant 55 
for 10 seconds, the information needed to reproduce the 
scene would be contained in one frame of the sequence 
and an indication of the number of times the frame is to 
be repeated. This would be approximately 1/300 of the 
information needed if the scene were sent without com- 60 
pression. 

To better understand the manner in which this type 
of prior art image compression operates, consider a 
motion picture which is being sent over a bandwidth- 
limited communication link from a transmitter to a re- 65 
ceiver. The receiver will be assumed to have a frame 
buffer which stores one frame of the motion picture. 
The contents of this frame are then displayed at the 



191 

2 

receiver. Ideally, the frame in the receiver will exactly 
match the corresponding frame at the transmitter. 

Typically, the process is started by sending one com- 
plete frame of the sequence. This frame is referred to as 
the key frame. After the key frame is sent, successive 
frames are specified by sending information specifying 
changes that have occurred relative to the last frame 
displayed by the receiver. The receiver uses this infor- 
mation to update its frame buffer. If the interframe 
changes are relatively small, i.e., the scene is almost 
constant, the available bandwidth will be sufficient to 
transmit all of the information needed to update the 
receiver's image. That is, the frame buffer in the re- 
ceiver will exactly match the corresponding frame of 
the motion picture at the transmitter. 

Unfortunately, the available bandwidth is often insuf- 
ficient to completely specify all of the interframe 
changes that have occurred since the last update in 1/30 
of a second. Hence, the changes to be sent are priorit- 
ized and only the highest priority changes are sent. As 
a result, errors accumulate in the frame buffer at the 
receiver. The transmitter keeps track of the magnitude 
of these errors. 

When these errors become sufficiently large, a new- 
key frame must be sent. The time needed to send a key 
frame is greater than 1/30 of a second. Hence, each time 
a key frame is sent, the motion will be interrupted for 
some period of time. For example, at a resolution of 
512x512 pixels each requiring 8 bits, a key frame would 
require between one and two seconds to send at 1.44 
Mbps, a data rate typically used in teleconference sys- 
tems. During this time interval additional frames will 
have been generated at the transmitter. The corrections 
needed to bring the receiver frame buffer up to date 
must be sent as a single large update. The combination 
of the time delay and the large update results in a notice- 
able loss of continuity in the motion. 

The various errors and motion interruptions become 
particularly objectionable when the sequence of scenes 
in the motion picture change rapidly. To minimize these 
problems, either the bandwidth of the communication 
channel must be increased or the quality of the picture 
must be reduced. Providing increased bandwidth in- 
creases the cost of transmitting the motion picture, and 
hence, is to be avoided. Similarly, a reduction in picture 
quality is also to be avoided. 

A second problem with this type of prior art motion 
picture compression system occurs in systems using 
interlaced video. In interlaced video systems, successive 
frames are offset relative to one another by an amount 
to one half the horizontal line spacing in a frame. Hence, 
the frames do not "line up" precisely when considered 
as sequence. Prior art systems do not provide an accept- 
able method for compensating for this motion. As a 
result, the decompressed motion picture often appears 
to have jitter. 

Broadly, it is an object of the present invention to 
provide an improved apparatus and method for com- 
pressing motion pictures. 

It is a further object of the present invention to pro- 
vide a method and apparatus which is less sensitive to 
scene changes than prior art systems. 

It is a still further object of the present invention to 
provide a method and apparatus which is more immune 
to compression artifacts in interlaced video systems 
than prior art systems. 
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These and other objects of the present invention will 
become apparent to those skilled in the an from the 
following detailed description of the invention and the 
accompanying drawings. 

SUMMARY OF THE INVENTION 5 

The present invention includes both apparatuses and 
methods for compressing a motion picture. An appara- 
tus for compressing a motion picture according to the 
present invention operates on a motion picture compris- 10 
ing a sequence of frames, each said frame comprising a 
two-dimensional image representing said motion pic- 
ture at one instant in time. The apparatus includes cir- 
cuitry for receiving a signal specifying a compression 
ratio for said motion picture and for receiving signals 15 
specifying said frames. A predetermined number N/ of 
consecutively received frames are stored by the appara- 
tus. These frames are processed by a three-dimensional 
filter which generates therefrom a plurality of three-di- 
mensional component images. The plurality of three-di- 20 
mensional component images includes a low-frequency 
component image and one or more high-frequency 
component images. Each of the component image com- 
prises a plurality of pixel values. The compression appa- 
ratus also includes quantization circuitry for approxi- 25 
mating each of the pixel values in at least one of said 
component images by an approximation having one of a 
predetermined set of integer values. The precision of 
said integers depends on the statistical distribution of 
the pixel values in said component image and said re- 30 
ceived compression ratio. These pixel approximations 
specify an approximation to the original N# frames of the 
motion picture. In the preferred embodiment of the 
present invention, the pixels of the low-frequency com- 
ponent image are approximated to a precision not less 35 
than the statistically optimum precision determined by 
said statistical distributions of said pixels in the compo- 
nent images and the received compression ratio and the 
pixels of at least one of said high-frequency component 
images are approximated to a precision not greater than 40 
the statistically optimum precision determined by the 
statistical distribution of the pixels in the component 
images and the received compression ratio. 

In the embodiments of the present invention in which 
color motion pictures are compressed, additional com- 45 
pression is obtained by compressing the chrominance 
signals to a greater degree than the luminance signals. In 
addition, the pixels of the chrominance signals are quan- 
tized and/or coded using vector techniques. 

By selectively retaining certain high-frequency infor- 50 
mation, significantly greater compression is obtained 
for motion pictures in an interlaced format. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a block diagram of a video transmission 55 
utilizing the compression technique of the present in- 
vention. 

FIG. 2 is a block diagram of an apparatus for filtering 
each frame of a motion picture to generate a plurality of 
two-dimensional component images. 60 

FIG. 3 illustrates the manner in which the apparatus 
shown in FIG. 2 may be iterative! y applied to generate 
additional two-dimensional component images of hav- 
ing different frequency content. 

FIG. 4 illustrates the manner in which a two-dimen- 65 
sional filter and a one-dimensional filter are combined 
so as to generate the three-dimensional filtered compo- 
nent images utilized by the present invention. 



FIG. 5 is a block image compression apparatus ac- 
cording to the present invention. 

FIG. 6 illustrates the manner in which a color motion 
picture is compressed by an apparatus according to the 
present invention. 

FIG. 7 illustrates the manner in which a single frame 
of a non-interlaced motion picture is filtered. 

FIG. 8 illustrates the manner in which a single frame 
of an interlaced motion picture is filtered. 

FIG. 9 illustrates the manner in which a motion pic- 
ture that has been compressed by the method of the 
present invention is decompressed. 

DETAILED DESCRIPTION OF THE 
INVENTION 

To simplify the following discussion, it will be as- 
sumed that a monochrome motion picture is to be com- 
pressed. The manner in which information specifying 
color is compressed will be discussed afterward. The 
compression technique of the present invention may be 
more easily understood with reference to FIG. 1 which 
is a block diagram of a video transmission apparatus 
utilizing the compression technique of the present in- 
vention. The motion picture consists of a sequence of 
two-dimensional images shown at 11. Each image com- 
prises a two-dimensional array of intensity values 1,^, 
where the x and y values correspond to points on a 
two-dimensional grid. Each array represents the motion 
picture at some time t. Hence, the motion picture may 
be represented by a three-dimensional array At 
any given time, compression apparatus 13 stores and 
processes a predetermined number of consecutive 
frames of the motion picture. The stored frames are held 
in a buffer 12. The stored frames are considered to be a 
single three-dimensional image. 

A three-dimensional perfect reconstruction filter 14 
operates on the stored three-dimensional image. The 
filter may be viewed as creating a number of component 
images. Each component image has information corre- 
sponding to information in specific scales, which corre- 
spond approximately to specific spatial frequency bands 
in the original three-dimensional image. The filtered 
images may be recombined using an inverse filter to 
regenerate the original three-dimensional image from 
the component images provided the filter and its inverse 
have sufficient numerical accuracy. 

It should be noted that the filtering transformation 
described above does not in itself provide image com- 
pression. If the filtered component images are represent 
to the same precision as the original frames of the mo- 
tion picture, the number of bits needed to store the 
filtered component images is the same as that needed to 
store the original motion picture frames. The actual 
image compression results from coding operations per- 
formed on the different filtered component images. 

The process by which the intensity values in the 
filtered images are coded will be referred to as quantiza- 
tion. Quantizer 15 replaces each suitably normalized 
intensity value by an integer which approximates the 
normalized intensity value. The error introduced by this 
approximation will depend on the range of intensity 
values and the number of bits allocated to each integer. 
If the range of intensity values is large, the number of 
bits needed to adequately approximate the intensity 
values will be larger than would be the case if the range 
of intensity values were small. 

Consider the pixel values of one of the component 
images. Each of the pixel values is to be replaced by an 
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integer having a predetermined number P of bits. The 
integers in question are then transmitted in place of the 
individual pixel values. At the receiver, the inverse of 
the mapping used to assign the integer values to the 
pixel values is used to recover the original pixel values. 
There are at most 2 P distinct values that can be repre- 
sented; hence, if there are more than 2 P different pixel 
values in the sub-set of pixel values, at least some of the 
pixel values will not be exactly recovered. The goal of 
the quantization algorithm is to minimize the overall 
error rate with respect to a suitable measure of error. 
Given the statistical distribution of the pixel values and 
the average number of bits to be allocated to each pixel, 
the optimum mapping of the pixel values to the integer 
values in question may be calculated. Since such optimi- 
zation calculations are well known in the art, they will 
not be discussed in detail here. For more details of the 
procedures in question, the reader is referred to William 
K. Pratt, Digital Image Processing, pp. 141-46, John 
Wiley & Sons, Inc., New York, 1978, which is hereby 
incorporated by reference. 

The overall error in approximating a component 
image will depend on the statistical distribution of the 
intensity values in the component images and the degree 
of image compression needed. The ratio of the number 
of bits needed to represent the original three-dimen- 
sional image to the number of bits in the compressed 
image will be referred to as the image compression 
ratio. Once an image compression ratio is set. the total 
number of bits available for all of the component images 
is determined. The optimum assignment of the available 
bits to the various component images is then determined 
from the statistical properties of the component images. 
It has been found experimentally that the statistical 
distributions of the component images generated by the 
present invention are approximated by Laplacian distri- 
butions. Hence, the optimum allocation of the bits may 
be made from a knowledge of the variance of the pixel 
values in each component image. Schemes for optimally 
allocating bits are known to those skilled in the art, and 
hence, the procedures for converting the observed vari- 
ances to quantization levels will not be described in 
more detail. For the purposes of this discussion, a quan- 
tization scheme will be defined to be "optimum** if it 
provides the lowest distortion in the reconstructed 
image for a given compression ratio. For the purposes 
of this definition, the distortion is measured in terms of 
mean-squared-error between the frames of the original 
motion picture and those of the motion picture obtained 
after compressing and decompressing the original mo- 
tion picture. It will be apparent to those skilled in the art 
that other statistical measures of distortion between the 
original and reconstructed motion pictures may be used. 

In the preferred embodiment of the present invention, 
the various component images are quantized using a 
uniform quantization scheme. In such schemes, the in- 
tensity of each pixel is transformed into an integer 
which is equal to the intensity value in question divided 
by a step size. In general, the step size will vary between 
component images. In the preferred embodiment of the 
present invention, the step sizes used for the various 
component images are determined by assuming a Lapla- 
cian distribution for the pixel values. The optimum step 
sizes are then determined from the variances of the 
pixels values of the component images, the bits needed 
to represent the original image, and the desired com- 
pression ratio. 



The filtering of the three-dimensional image into 
component images of different spatial and time domain 
frequencies serves two functions in the present inven- 
tion. First, the filtering operation reduces the correla- 
5 tion of the pixels in space and time. This reduction in 
correlation leads to reduced variances, and hence, re- 
duces the number of bits needed to adequately represent 
the various component images. Most motion picture 
images are highly correlated both in space and time. 
10 Hence, the three-dimensional images generated from a 
number of consecutive frames will be highly correlated 
in all three dimensions. If the three-dimensional image 
can be transformed utilizing a linear transformation in 
space and time such that the transformed image has 
15 reduced correlation, then the variance of the pixels of 
the transformed image will be less than that of the origi- 
nal image. In this representation, relatively few of the 
pixels in the transformed image will contain most of the 
image signal energy, and the remaining pixels will have 
20 reduced variance and. hence, can be coded with fewer 
bits. The filtering operation utilized in the present in- 
vention provides this transformation. 

It is found experimentally that the lowest frequency 
component image has a variance at least as large as the 
25 original three-dimensional image; however, the various 
high-frequency images have significantly lower vari- 
ances. As will be explained in more detail below, the 
total number of pixels in the various component images 
is approximately the same as that in the original image. 
30 However, the number of pixels in the lowest frequency 
component image is a small fraction of the total pixels in 
the transformed image. Hence, even if pixels of the 
lowest frequency component image are quantized to an 
accuracy equal to or greater than that of the original 
35 image, the more numerous pixels of the various high- 
frequency component images can be quantized to an 
accuracy much less than that of the original image. As 
a result, a net image compression will be obtained. 
Second, the filtering operation separates the image 
40 information into subsets which have different subjective 
value to a human observer. The three-dimensional im- 
ages processed by the present invention have two spa- 
tial dimensions and one time dimension. It is known 
from the study of human vision that low spatial fre- 
45 quency information is more important to a viewer than 
high spatial frequency information when viewing still 
photographs. It is also known from the study of vision 
that the human eye cannot resolve fine detail in moving 
objects. Hence, objects in an image that are moving can 
50 be displayed with lower precision than stationary ob- 
jects. Thus, the time dimension behaves in the same 
manner as the spatial dimensions of the individual 
frames in that low frequency information is more im- 
portant than high frequency information. In the pre- 
55 ferred embodiment of the present invention, more bits 
are allocated to the low frequency component image 
than would be allocated using a statistically optimum 
quantization scheme. This allocation preferentially re- 
tains the information most important to a human ob- 
60 server. In addition, fewer bits per pixel are allocated, on 
average, to the high frequency component images than 
would be allocated by a statistically optimum quantiza- 
tion scheme. This allocation improves the image com- 
pression by selectively degrading the information 
65 which is less important to a human observer. In the 
preferred embodiment of the present invention, some of 
the highest frequency component images are not quan- 
tized at all. The non-quantized component images are 
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discarded. Since the number of pixels in the Jow fre- 
quency component image is a small fraction of the num- 
ber of pixels in the various high frequency component '«jw ~ ? ? J J ""C^^f^xj.D 
images, additional compression is obtained for a given 

degree of subjective image quality. 5 set of functions {-Fyjtfx.y.t)} forms a basis for a 

Referring again to FIG. 1, the component images, vector space. The various values of m correspond to the 

after quantization, are transmitted over a bandwidth different component images, and the coefficients "C v .* 

limited communication channel 16. At the receiving end are the pixels of the various component images, 

of the channel, the inverse three-dimensional filter is For compression applications, the basis func- 

apphed to reconstruct an approximation to the original 10 tions preferably have a number of additional properties, 

motion picture by a decompression apparatus 21. The Firsl . it is advantageous to utilize an approximation in 

bandwidth limited channel may include storage devices which the basis functions form an orthonormal basis. It 

such as magnetic disks or other devices of finite capac- can be shown that an orthonormal basis requires fewer 

ity. Hence, the method of the present invention may coefficients m djk to adequately represent the image 

also be used to store motion pictures in a form requiring 15 tha n a non-orthonormal basis having similar spatial 

less storage medium than would be needed in the ab- properties. Hence, a filter based on an orthonormal 

sence of compression. basis concentrates the information in component images 

Decompression apparatus 21 performs the inverse of having fewer pixels which, in turn, permits a higher 

the transformations carried out by compression appara- compression ratio for any given image quality, 

tus 13. The quantized component images are received 20 Second, it is advantageous to use a basis in which the 

by an inverse quantizer 19 which generates intensity basis functions have properties which permit efficient 

levels from the quantized values. Inverse quantizer 19 representation of image information found in the motion 

inserts zeros for the pixels of the component images that picture. If this property is satisfied, fewer coefficients 

were not quantized. The resulting three-dimensional 2J mQj.k will be needed to adequately represent the image 

component images will be approximations to the origi- than would be the case if basis functions having proper- 

nai three-dimensional component images produced by ties that are radically different from those of the objects 

filter 14. The intensity values in question are stored in a in the motion picture are used. For example, most im- 

buffer 18. An inverse perfect reconstruction filter 20 ages, and hence most motion pictures, are made up of 

then constructs a single three-dimensional image from 3Q objects whose intensity varies smoothly over the object, 

the component images. The single three-dimensional Hence, a basis in which the basis functions approximate 

image in question is then divided into a plurality of polynomials in the spatial dimensions is desirable. In 

frames which are then outputted as shown at 22. addition, most motion pictures have edges in the spatial 

In general, the three-dimensional filter operates on a dimensions and include scenes that are relatively con- 
three-dimensional image to generate a plurality of 35 slant in time for some period and then suddenly change, 
three-dimensional component images which may be Hence, it is desirable to utilize a basis which can ade- 
recombined by the inverse filter to regenerate the origi- quately approximate edges in space and time. It has 
nal image. The preferred filter creates a low-frequency been found experimentally that these properties can be 
component image and a plurality of high frequency adequately satisfied by using basis functions that ap- 
component images. The low-frequency component 4Q proximate low order polynomials. In particular, basis 
image will have an average pixel intensity value that is functions approximating quadratics or cubics in the 
greater than zero, whereas the average pixel intensity spatial dimensions are found to be optimum, 
value in the high frequency images will be zero. The Third, the spatial extent of each basis function should 
high-frequency component images preferably have be limited. The port ion of the image over which a given 
lower correlation than the original image. In addition, 45 basis function is non-zero is referred to as the support of 
the frequency content of the various high frequency the basis function. If the support of each of the basis 
images is preferably different for different sets of high- functions m Fjj,k (x,y,t) is small compared to the image 
frequency component images. That is, the collection of being approximated, an error in the corresponding coef- 
high-frequency component images may be organized ficient m C//* introduced by the quantization process 
into groups having component images with different 50 will only affect a small fraction of the reconstructed 
frequency content. Within a group, different compo- image. In this case, random errors in the coefficients 
nent images may emphasize different features of the will result in noise-like errors in the reconstructed im- 
images. For example, one component image may con- age. If, however, the support of the basis functions is 
tain information relating to horizontal lines in the im- large, a quantization error in a coefficient will lead to an 
age, while a second component image may have the 55 error in the reconstructed image that affects a large 
same frequency content but contain information relat- fraction of the image. Such errors may appear as stripes 
ing to vertical edges. That is, the high frequency com- or other artifacts. It is known from physiological re- 
ponent image emphasizing horizontal edges would con- search that noise-like errors are far less objectionable to 
tain a line segment at the location of a horizontal edge if a human observer than coherent artifacts such as stripes, 
the filter preserves local structure. 60 Hence, it is advantageous to utilize a filter that corre- 

Finally, it is advantageous to choose filters that are sponds to a functional expansion employing basis func- 

sensitive to the types of spatial and time variations that tions having limited support. 

are normally encountered in motion pictures. It may be The manner in which the perfect reconstruction filter 
shown that the filtering operation described above is and its inverse are constructed will now be explained in 
equivalent to approximating the original image by a 65 more detail. In the preferred embodiment of the present 
linear superposition of basis functions. That is, an image invention, three-dimensional filter 14 is constructed by 
represented by an array l x ^ may be approximated by a iteratively applying a one-dimensional perfect recon- 
venes of the form struction filter to the three-dimensional image. The 
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preferred filter is a conjugate mirror filler bank that 
generates two outputs from an input signal comprising a 
linear sequence of signal values. The outputs in question 
comprise low-frequency and high-frequency compo- 
nents of the input signal. In the preferred embodiment 
of the present invention, the filter bank is implemented 
as digital finite impulse response filters (FIR). Hence, 
the conjugate mirror filter bank accepts a vector having 
M components. x(i). If M is not even, a zero is appended 
to x to force M to be even. The filter bank multiplies the 
elements of x with the elements of a vector having 
components a(k), where k runs from 0 to N - 1, to form 
two vectors having components, y^O) and y//(i). These 
vectors each have M/2 such elements. The components 
of yz.(i) and y/y(i) are calculated as follows: 



10 



.v- 1 

2 (-]) k k m a{k) = 0 
fc=0 



(4a) 



for m= 0.1. .... D where D is the degree of the desired 
polynomial approximation. An exemplary set of filter 
coefficients having N = 6 is as follows: 
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k 


a(k> 


0 


.3327 


1 


.8060 


2 




3 


-.1350 


4 


-.0854 


5 


.0352 



>X(2m) = ' 1 a(k)x(2m - X - k) 
k~0 

yH (2m T l) = 4ft) (-1/- 1 x(lm + 1 - k) ° J 20 

k=Q 

Here, m runs from 0 to (M/2) — I. 

The transformations shown in Eqs. (1) and (2) require 
that some value be assigned to pixels that lie outside the 25 
signal range, i.e., x(i) for i<0 or i>M - 1. In the pre- 
ferred embodiment of the present invention, x is as* 
sumed to be cyclic, i.e., x(i)=x(i+M). In this case, it 
can be shown that yz. and y// each have at most M/2 
non-zero components, and the total number of pixels in 30 
the various component images will be the same as that 
in the original image. Other possible assumptions for the 
values outside the defined range of x(i) will be apparent 
to those skilled in the an. For example, the values out- 
side the range could be set to zero. In this case, the 35 
maximum number of non-zero components will be ap- 
proximately M/2. 

The coefficients a(k) shown in Eqs. (1) and (2) are 
real or complex numbers that must satisfy the following 
relationship for the filter bank to be a conjugate mirror 40 
filter bank: 

.v- i O) 

Z oM)o<i+ k) = 26io 

'=° 45 

Here, a(k)=0for k<0 or k>(N- 1). and k is any even 
integer, and a *(i) denotes the complex conjugate of the 
(possibly) complex number a(i). It may be shown that 
filters satisfying Eq. (3) are equivalent to expanding the 50 
input signal x(i) in a orthonormal set of basis functions. 
It can be shown that the three-dimensional filter ob- 
tained by the iterative application of such a one- 
dimensional filter is equivalent to expanding the three- 
dimensional image in a three-dimensional orthonormal ^ 
basis of the type discussed above. In addition, for the 
filter to generate high and low frequency components, 
the following relationship must be satisfied. 

(4)60 

It may be shown that the support of the basis functions 
is determined by the number of a(i) which are non-zero. 
Hence, filters having a small number of weights are 65 
preferred. Filters in which the weights satisfy Eqs. (3) 
and (4) will provide good polynomial approximations if 
they also satisfy the following system of equations 



In the preferred embodiment of the present invention, 
each frame of the motion picture to be included in a 
particular three-dimensional image is first filtered to 
form a plurality of two-dimensional filtered component 
images. Corresponding two-dimensional filtered com- 
ponent images in a plurality of these frames are then 
filtered to form the plurality of three-dimensional fil- 
tered component images discussed above. 

The manner in which each frame is coded as a plural- 
ity of two-dimensional filtered component images will 
now be explained with reference to FIGS. 2 and 3. 
Each frame of the motion picture is assumed to be rep- 
resented by an array of pixel values having M columns 
and M rows. The generalization of the method taught 
below to non-square arrays of pixels will be apparent to 
those skilled in the art from the following discussion. 
Each frame is coded by iteration of a two-dimensional 
filtering transformation. The two-dimensional transfor- 
mation is carried out by applying a one-dimensional 
transformation twice as indicated in FIG. 2. In the first 
step, an image 42 comprising a single frame of the mo- 
tion picture with M columns and M rows of pixels is 
transformed into two images denoted by L and H by a 
filter circuit 44 which will be referred to as analyzer 44. 
Analyzer 44 comprises a conjugate mirror filter bank as 
described above. The L and H images each have (M/2) 
columns and M rows. Each row in the L and H images 
is generated by filtering the corresponding row in the 
image 42 through analyzer 44. The high frequency 
output of analyzer 44 becomes the corresponding row 
in the H image, and the low frequency output of analy- 
zer 44 becomes the corresponding row in the L image. 

In the second step in the transformation, each of the 
L and H images is funher transformed into two 
(M/2) X (M/2) component images. Image L is used to 
create two images, LL and LH in a manner analogous 
to that described above, except that the image is divided 
into columns instead of rows. Each image is divided 
into its M element column vectors. Each vector is trans- 
formed using Analyzer 44 into two M/2 element col- 
umn vectors having elements y^and y# using the trans- 
formation defined in Eq. (1). The vector y± obtained by 
transforming the \ th column of image L becomes the i' A 
column of image LL. Similarly, the vector y// becomes 
the \ th column of image LH. The remaining images HL 
and HH are obtained in an analogous manner. 

Upon completion of the transformation, the original 
frame will have been filtered into four two-dimensional 
component images having differing spatial frequency 
characteristics. The LL component image corresponds 
to the component image representing the low frequency 
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spatial information. The LH component image empha- 
sizes horizontal high frequency information, such as 
horizontal edges. The HL component image empha- 
sizes vertical high frequency information, and the HH 
component image emphasizes high frequency informs- 5 
tion related to edges at 45 degrees and 135 degrees. 

As noted above, human observers place more impor- 
tance on the low frequency information, i.e., the LL 
component image. If only this component image were 
retained, and the pixels thereof coded using the same 10 
number of bits per pixel as used to code the original 
frame of the motion picture, an image compression of 
4:1 would be obtained. Unfortunately, 4:1 compression 
ratios are insufficient for most purposes. Further, as will 
be discussed in more detail below, retention of at least IS 
some of the high frequency information is needed to 
maintain picture quality. 

To increase the compression ratios, the transforma- 
tion described with reference to FIG. 2 is iterated on 
the LL image as illustrated in FIG. 3. It will be assumed 20 
that the original image 72 has PxP pixels. Image 72 is 
decomposed into four component images, 74-77, as 
described above, component image 74 being the LL 
image. Each of the component images 74-77 are 
(P/2)X(P/2) pixels. 25 

Next, component image 74 is decomposed into its 
four component images 80-83, component image 80 
being the LL component image. Component images 
80-83 each have (P/4) x(P/4) pixels. The iteration con- 
tinues by decomposing the LL component into its com- 30 
ponent images at each stage. Hence, at the next stage, 
component image 80 will be decomposed into compo- 
nent images 84-87. The iteration is continued until the 
LL image is reduced to some predetermined size. 

The frequency characteristics of the various compo- 35 
nent images will depend on the specific filler coeffici- 
ents used. In addition, the frequency characteristics will 
depend on the level of the decomposition at which ihe 
component image was generated. For example, compo- 
nent images 75-77 will in general represent higher fre- 40 
quency information than component images 81-83 
which in turn will represent higher frequency informa- 
tion than component images 85-87. 

It should be noted that the memory space needed to 
store the image at each stage of the decomposition is the 45 
same as that needed to store the original image. While 
the device shown in FIG. 2 utilized separate memory' 
arrays for the various component images at the different 
stages of the transformation, it will be apparent to those 
skilled in the art that the outputs of analyzer 44 can be 50 
placed back in the same array used to store the original 
frame pixel array provided the outputs in question are 
appropriately buffered. 

Once the single frames making up a given three-di- 
mensional image have been filtered as described above, 55 
the filtered two-dimensional component images are 
combined to generate a set of three-dimensional compo- 
nent images representing the three-dimensional image. 
An apparatus for generating the three-dimensional fil- 
tered component images is shown in FIG. 4 at 100. 60 
Coding apparatus 100 receives a sequence of frames 101 
which provide the input to a two-dimensional filter 102. 
Filter 102 operates iterative! y on each frame to generate 
a set of two-dimensional filtered images in a manner 
analogous to that described above with reference to 65 
FIGS. 2 and 3. The output of filter 102 for each frame 
is stored in a time frame buffer. Exemplary time frame 
buffers are shown at 104-107. The component images in 
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each time frame buffer are stored such that correspond- 
ing pixels are stored at like addresses. A sequence of 
such pixels is shown at 111-114. 

Each sequence of corresponding pixels in the time 
frame buffers is treated as a one-dimensional digital 
signal. The digital signal is inputted to a one- 
dimensional filter 121 which is similar to analyzer 44 
shown in FIG. 2. The output of filter 121 comprises the 
pixels of high and low frequency filtered three-dimen- 
sional images. These pixels are stored in a three-dimen- 
sional buffer 122. Filter 121 may be applied iteratively 
to the contents of buffer 122 to provide three-dimen- 
sional component images with different degrees of com- 
pression in the time-domain in a manner analogous to 
that described with reference to FIG. 4 above. While a 
separate buffer 122 is shown for storing the three-di- 
mensional component images, it will be apparent to 
those skilled in the art that the three-dimensional com- 
ponent images may be stored back in the time compo- 
nent images, may be stored back in the time frame buff- 
ers in a manner analogous to that described above with 
reference to the buffers shown in FIG. 2. 

The amount of memory needed to store the three-di- 
mensional component images is determined by the num- 
ber of taps in filter 121, i.e., the value of N in Eq. (1), 
and the number of times the time transformation is iter- 
ated. If filter 121 is only applied once, i.e., the transfor- 
mation is not iterated, the number of frames needed for 
the three-dimensional image will be N. Since the mem- 
ory need for storing the three-dimensional component 
images represents a significant fraction of the hardware 
expense, time-domain transformations with a small N 
value are preferred. The smallest value of N is 2. 

In the preferred embodiment of the present invention, 
the filtration and quantizing operations are carried out 
by a pipe-lined processor. The operation of such a pro- 
cessor 200 will now be explained in detail with refer- 
ence to FIG. 5. The individual frames 201 of a motion 
picture are received by image compressor 200 one 
frame at a time. Each time a frame is received from the 
motion picture, the frame is immediately filtered in two 
dimensions to form a set of two-dimensional filtered 
component images. The two-dimensional component 
images are stored in a time buffer 204. It will be assumed 
that the time domain filter requires N, frames from the 
motion picture to construct the three-dimensional fil- 
tered picture. N, will in general be the number of a(k) 
values in the one dimensional time domain filter. Hence, 
the number of time domain buffers, must be at least N,. 

The second stage of the. pipeline requires groups of 
N, frames as input. When the two-dimensional compo- 
nent images from N r frames have been accumulated, the 
controller 206 causes the one-dimensional time filter 208 
to generate the three dimensional component images 
210. In the preferred embodiment of the present inven- 
tion, these component images are stored in the time 
buffers in which the two-dimensional component im- 
ages were stored. However, to clarify the present dis- 
cussion, a separate buffer is shown in FIG. 5. During 
this process, incoming frames that have been trans- 
formed in two dimensions are stored in one or more 
additional time buffers. The number of additional time 
buffers depends on the time needed to complete the 
time domain filtering, quantize the pixels of the filtered 
three-dimensional component images, and transmit the 
quantized pixels on the communication link. There must 
be sufficient buffers to hold all of the two-dimensional 
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filtered component images generated during these oper- of the present invention and a luminance/chrominance 

at ions. format. 

The third stage of the pipeline computes the statistics In a luminance/chrominance format, each frame is 

of the pixel groups to be quantized. In the preferred also represented by three frames. The first frame repre- 

embodiment of the present invention, controller 206 5 sents the overall light intensity associated with each 

includes data specifying which pixels are to be exam- pixel of the frame. The remaining two frames, referred 

ined for possible quantization. In general the pixels may to as the chrominance frames, provide color signals 

be viewed as comprising three groups. One group will which when combined with the luminescence frame 

always be quantized. The low-frequency component generate the RGB frames described above. It is found 

image is such a group. The low-frequency component 10 experimentally that significantly more compression can 

image is typically quantized to a predetermined accu- be applied to the chrominance frames than to the lumi- 

racy that is set independent of the statistics of this com- nance frame - Viewed separately, the chrominance 

ponent image. This accuracy is sufficiently high to guar- fram « would appear of a much poorer quality than the 

antee that quantization errors in this component image luminance frame; however, whenrecombined to form 

will be negligible. 15 RGB frames at the receiver, the lower quality of the 

The second group is always ignored. That is. zero bits color chrominance frames is not apparent to a human 

are allocated to the pixels of this group. With the excep- observer. As a result, significantly greater overall com- 

tion of the high frequency component image discussed P rcss,on «■> ** accomplished while preserving the ap- 

below with reference to horizontally scanned and inter- P arem "jagc quality of the motion picture. Hence, the 

laced data, the highest frequency component images are 20 preferred embodiment of the present invention operates 

tvoicallv ifinored " on a l uminancc frame and two chrominance frames. If 

Finally, there are groups of component images that the motion P. icture is . n ?' in this "PPropriate 

mav be quantized if the group has sufficient power in «™ers.on circuitry is incorporated to generate the 

the component image. Here, the power in a component „ ' uminan ff -«» chrominance frames from the RGB 

image is defined to be the sum of the squares of the pixel 2J f ™ e *[ ?* mann f r ,n , whl( * *e conversion is accom- 

values for that image. This last class may also include P ,,shed "conventional ,„ the photographic arts, and 

. r e » * * hence, will not be described in more detail here, 

portions of a component image. In the larger compo- Th£ , inven , jon ^ make& uje f re , atjon . 

nent .mages, it is sometimes advantageous to a.v.de the ^ ^ tween , he data 

in the two chrominance frames 

component image into blocks which are quantized sepa- }Q tQ ' further ^ the chrominance data . ,„ 

rate y. 1 hese larger component images represent rela- , he e of co j ors 

in an imaee is considerablv less than 

lively high frequency information. Only those portions ^ maxijTlurn range that would be deduced from the 

of the component image corresponding to edges in the variations in lhe chrom inance images. Images are typi- 

original image are significant. If the entire component ca „ v madc of objccts of reIativcIy uniform colors< 

image were to be quantized, a large number of bits 35 Con - sider a uni f onn ] v colored object. There will be 

would be devoted to the regions having no useful infor- corresponding uniform intensity objects in each of the 

mation. If. however, only those regions having a signifi- chr ominance frames in the same locations. Hence, the 

cant number of non-zero pixels, or some other discern- data in the two chrominance frames will be highly cor- 

able feature such as a line, are quantized, this waste can related in the regions occup i ed b v the object in ques- 

be reduced. The number of bits allocated to this group 40 tion . This corre lation may be used to further reduce the 

of component images depends on the statistics of the num ber of bits needed to store the chrominance infor- 

pixels in each component image and frequencies repre- mation. 

sented by each component image. As noted above, FIG. 6 illustrates the compression of a motion picture 

higher frequency component images may be quantized comprising luminance frames 401 and first and second 

utilizing less than the statistically optimum number of 45 chrominance frames 402 and 403 by a compression 

D * ts - apparatus 400 according to the present invention. The 

Once the three-dimensional component images have luminance frames 40! are compressed in a manner anal- 
been quantized, the information specifying the quantiza- 0 gous to that described above with reference to FIG. 5. 
tion transformation is sent over the communication link A s each luminance frame 401 is received, a two-dimen- 
followed by the quantized pixels. In the preferred em- 50 sional filter 410 decomposes the frame into two-dimen- 
bodiment of the present invention, the quantization sional component images. The component images cor- 
mformation is sent first to enable the receiver to reverse responding to each received luminance frame are stored 
the quantization of the pixels as the quantized pixels are i n the next available frame butTer 414. When a predeter- 
received. mined number of luminance frames have been so pro- 

The above described embodiments of the present 55 cessed. a time domain filter 418 converts the two-di- 
invention have been discussed with reference to mono- mensional component images to a plurality of three-di- 
chromatic motion pictures. The manner in which color mensional component images which are stored in a 
motion pictures are compressed will now be discussed. buffer 420. The three-dimensional component images in 
Color may be specified either in a RGB format or in a buffer 420 are then quantized by quantizer 422 to form 
luminance/chrominance format. In the RGB format 60 a luminance output signal 424. 
three frames are specified for each frame of the motion The first and second chrominance frames 402 and 403 
picture, one giving the intensity of red, one giving the corresponding to the luminance frames received by 
intensity of blue, and one giving the intensity of green. filter 410 are processed by two-dimensional filters 411 
In principle, each of these frames may be separately and 421 in an analogous manner to generate two-dimen- 
compressed and transmitted over the communication 65 sional chrominance component images which are stored 
link and decompressed at the receiver. However, the m frame buffers 424 and 434. In the preferred embodi- 
amount of compression obtained is significantly less ment of the present invention, only the low-frequency 
than that obtained utilizing the preferred embodiment component images which are shown at 425 and 435 are 
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actually generated and subjected to the subsequent pro- relating to horizontal edges, component image 232 em- 

cessing. When the above mentioned predetermined phasizes high frequency information relating to vertical 

number of frames have been received, time-domain edges, and component image 231 emphasizes high fre- 

filters 419 and 439 convert the two-dimensional chromi- quency information related to diagonal edges. Nor- 

nance component images to three-dimensional compo- 5 ma ]j y> thesc component images would be discarded or 

nent images which are stored in buffers 428 and 448. In al mosl C€rtain bIocks of pixels therein quantized to 

the preferred embodiment of the present invention, only limited precision 

the low-frequency chrominance three-dimensional When |hfi ^ CQnsisls of imerlaced ^ dala , 

component images shown at 417 and 437 are actually nent , m corresp onding to edges in the 

generated and subjected to the subsequent processing. 10 £ f £ f fa fa the 

The degree of compression applied to the first chro- . , , . . ru ~« a ~r 

minance frames is the same as thai applied to the second f eri ? ced data ' » r,lter i?L and a P° mon °f 

chrominance frames; hence, the component images ! he data retained. Referring to FIG. 8. component 

stored in buffers 428 and 448 have corresponding pixels. ima S e 230 ,s fillcred ,n two-dimensions in a manner 

Each pair of corresponding pixels may be considered to 15 analogous to the filtering of the first low-frequency 

be components of a two-dimensional vector. That is, a component image 234 shown in FIG. 6. That is, compo- 

single vector valued component image is created from nent image 230 is input to a two-dimensional filter and 

each of the corresponding component images in buffers the resulting four component images stored back in the 

428 and 448. These vector valued component images same portion of the two-dimensional array as originally 

are then quantized by vector quantizer 438 to form a 20 used for storing component image 230. The resulting 

chrominance output signal 444. Because of the above- four component images are shown at 240-243 of FIG. 7. 

mentioned correlation between the first and second The process is iterated on the low-frequency compo- 

chrominance frames, the number of vector values actu- nent image generated by each filtration pass. At the 

ally realized in the motion picture is considerably less completion of the process, the original image will have 

than the product of the numbers of values realized in 25 been decomposed into two hierarchical sets of compo- 

each chrominance frame. As a result, the number of bits nent images and the ori gj na l high frequency component 

needed to represent the chrominance data is signifi- images shown at 231 and 232 in FIG. 7. 

cantly reduced. . . . , In the preferred embodiment of the present invention, 

Before describing the manner in which motion pic- co onent ima s m and 232 are not quanti2ed at all. 
tures compressed by the present invention are decom- 30 Thcsc c0 nt ima arc discardcd . compo _ 
pressed, the manner in which interlaced motion picture r , . , f „ ft , \_ 
images are preferably compressed will be described. An " en ' L™ 8 * dmVed fm com P on ™l™*& ™ and tne 
interlaced motion picture may be regarded as a motion "^-frequency component images derived from corn- 
picture with frames having half the number of horizon- P° nem ima S e 234 are quantized to a precision that de- 
tal scan lines in which every other frame is offset by half 35 P ends on the v *nance of the particular component 
the line spacing in the vertical direction. In effect, each ima S e and the original component image.from which it 
frame fills in the spaces between the lines of the previ- was derived. The component images derived from com- 
ous frame. In principle, each of the frames may be ponent image 230 are quantized to a precision which is 
treated as individual frames of the motion picture and less than the precision that would be dictated by an 
compressed as described above. However, a significant 40 optimum quantization algorithm. The high-frequency 
improvement in compression can be obtained if the component image derived from component image 234 
frames are combined to form a single frame having are quantized at the optimum level. The low-frequency 
twice the number of scan lines which is then com- component image derived from component image 234 is 
pressed. quantized to a much higher precision than that dictated 

The amount of compression that may be obtained is 45 by an optimum quantization algorithm. 

dependent on the amount of redundant information in a The motion pictures compressed by the compression 

frame. There is considerably more redundant informa- method of the present invention can be decompressed 

tion in successive frames of an interlaced sequence than by reve rsing the quantization steps and then combining 

in a single frame; hence, improved compression may be the varjous flhered three-dimensional component im- 

obtamed. Unfortunately, it is found experimentally that 50 agcs utnJzj thc invcrsc of thc FIR fihcr banks uscd t£> 

merely combining two successive frames and compress- ate tne component imagcs< Sincc the inversc fi|ter 

ing the combined frame leads to unacceptable artifacts. f an . c ^ 0 t ' * t • „^ _ a ■ . 

1 . . - » .» — ■ .v. . v . * ■ banks combine two component images to form a single 

In particular, certain fine-grained background which is • _ # . m u u 11 r -1* 

«t J;™*™ j« #k- ^««;«oi * n image, the inversc filter banks will be referred to as 

stationary in the original motion picture sequence ap- *f 

pears to vibrate back and forth in the decompressed 55 svnln " ,zers ' 

motion picture. It has been found experimentally that u In the P refcrrcd embodiment of the present invention, 

this artifact can be alleviated by preferentially quantiz- thc » accomplished by iteratively applying 

ing certain high frequency information that would oth- one-dimensional synthesizers. A one dimensional syn- 

erwise not be quantized at all or at most only with lim- thesizer operates on the high and low frequency vectors 

ited accuracy. 60 generated by the conjugate mirror filter bank described 

Consider the two-dimensional decomposition of a abovc to form a si S nal vcctor whose hi gh and fre- 

frame into two-dimensional filtered component images quency components are the high and low frequency 

as described above with reference to FIG. 3. After vectors in question. If the components of the low and 

decomposition, the two-dimensional component images high frequency vectors are denoted by xlO) and xhQ), 

are organized into a hierarchy as shown in FIG. 7. The 65 respectively, the synthesizer generates an output vector 

first level of decomposition generates three high fre- having components z(i). The number of components in 

quency component images shown at 230-232. Compo- the output vector is twice that in each of the input vec- 

nent image 230 emphasizes high frequency information tors. The output vector is calculated as follows: 



3/15/05, EAST Version: 2.0.1.4 



5,121,191 

17 18 

It should be noted that additional compression of the 
v .-i_l (5) quantizer output in the present invention may be ob- 

rOn = ^ Q oiiU\ L [m - i.V2> - k] - " tained by using conventional arithmetic coding schemes 

to eliminate any redundancy in the quantizer output, 
.v.? -I 5 Such schemes replace repeated sequences of values by 

aEo <rf2A ~ " codes requiring fewer bits than the sequence in ques- 
tion. It is to be understood that the quantizers and deq- 
. , V/ J- 1 (6) uantizers described above may also include such coding 

fc=o schemes. Vector coding schemes are also known to 

10 those skilled in the art and may be applied to the vectors 
oiikUHim - k) generated from the chrominance component images. 
*°° As pointed out above, the bandwidth limited commu- 

nication channel may include some means for storing 
Here, the a(k) are the filter coefficients used by the filter lne compressed motion picture such as magnetic tape or 
bank that generated the high and low frequency vectors j 5 computer disk storage. Hence, the present invention 
\jf and xi;. may be utilized to generate a recording of a motion 

As noted above, the decompression process is begun picture in a compressed format. Such recordings may be 
by first reversing the quantization transformation. If a duplicated in the compressed format and disseminated 
particular component image was discarded prior to to end users equipped with the appropriate decompres- 
quantization, the component image in question is re- 20 sion apparatuses. 

placed by a component image of the appropriate size in Such a recording would include means such as a 
which all of the pixel values are zero. Since the various magnetic or optical storage medium for storing a plural- 
component images were quantized to differing degrees j ty 0 f digital values. The digital values would be divided 
of precision, the resultant filtered three-dimensional into groups in which each group would include approx- 
component images are only an approximation to the 25 imations for the pixels of one or more component im- 
three-dimensional component images generated from ages, each such component image specifying a three-di- 
the sequence of frames received by the transmitter. mensional image. Each of the three dimensional image 

As apparatus for decompressing the motion picture is would include information specifying two or more 
shown at 300 in FIG. 9. Approximations to three-di- frames of the motion picture. The component images 
mensional component images are generated from the 30 according to this embodiment of the present invention 
quantized data received by dequaniizer 302 which re- would include a low-frequency component image and 
ceives data from the bandwidth limited communication one or more high-frequency component images. In the 
channel 303. The three-dimensional component images preferred embodiment of this recording, each group 
are stored in a buffer 304. The filtering transformation would also include digital values specifying two-dimen- 
are reversed using the appropriate synthesizers to gen- 35 sional chrominance vectors as described above. If the 
erate a sequence of frames which approximates the recording had been derived from an interlaced scanning 
frames used to construct the original three-dimensional device, each group would also include a high frequency 
image. First, a one-dimensional time domain synthesizer component image emphasizing horizontal image fea- 
is used 10 generate a plurality of two-dimensional filter tures in the frames of said motion picture, 
component images which are stored in time buffers 306. 40 There has been described herein a novel apparatus 
The time buffers can be the same as the buffers used to and method for compressing and decompressing motion 
store the three-dimensional component images. How- pictures. Various modifications to the present invention 
ever, for clarity, separate buffers are shown. Each time will become apparent to those skilled in the an from the 
buffer 306 stores the two-dimensional filtered compo- foregoing description and accompanying drawings, 
nent images needed to generate one frame of the motion 45 Accordingly, the present invention is to be limited 
picture after the time domain synthesizer has acted. The solely by the scope of the following claims, 
contents of each buffer are then iteratively passed What is claimed is: 

through synthesizer 310 which combines the various 1. An apparatus for compressing a motion picture 
component images into a single two-dimensional image comprising a sequence of frames, each said frame com- 
which is outputted from apparatus 300. 50 prising a two-dimensional image representing said mo- 

Apparatus 300 is preferably constructed as a pipe- tion picture at one instance in time, said apparatus corn- 
lined processor. In the first stage of the processor, de- prising: 

quantization takes place and the three-dimensional fil- means for receiving a signal specifying a compression 
tered component images generated. The second stage of ratio for said motion picture; 

the processor reverses the time domain transformation. 55 means for receiving signals specifying said frames; 
The third stage of the processor generates a motion means for storing a predetermined number, N, of 
picture frame from each of the sets of two-dimensional consecutively received said frames; 

component images. three-dimensional filtering means for filtering said 

The apparatus shown in FIG. 9 may be replicated for stored frames to generate a plurality of three-di- 

the purpose of decompressing color images that have 60 mensional component images, said plurality of 
been compressed by the method described above. In three-dimensional component images comprising a 

this case, the apparatus shown in FIG. 9 would be used low-frequency component image and one or more 

to decompress the luminace image. The version used to high-frequency component images, each said corn- 

decompress the chrominance images would contain two ponent image comprising a plurality of pixel val- 

such apparatuses and a dequantizer that unpacks the 65 ues; and 

quantized vector data placing the different components quantization means for approximating each pixel val- 
of the vectors in the buffers analogous to buffer 304 ues in at least one of said component images by an 

described with reference to FIG. 9. approximation comprising one of a predetermined 
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set of integer values, the precision of said integers 
being related to the statistical distribution of the 
pixel values in said component image and said re- 
ceived compression ratio; and 
means for generating output signals specifying said 5 
pixel approximations. 

2. The motion picture compression apparatus of claim 
1 wherein said quantization means comprises means for 
approximating the pixels of said low-frequency image to 

a precision greater than the statistically optimum preci- 10 
sion determined by said statistical distributions of said 
pixels in said component images and said received com- 
pression ratio; and means for approximating said pixels 
of at least one of said high frequency component images 
to a precision less than the statistically optimum preci- 15 
sion determined by the statistical distributions of said 
pixels in said component images and said received com- 
pression ratio. 

3. The motion picture compression apparatus of claim 

1 wherein said frames of said motion picture comprise 20 
interlaced frames, every other said frame being offset by 
one half the horizontal line spacing of said frames, 
wherein said receiving means comprises means for com- 
bining each pair of consecutively received frames to 
form a single frame having twice the number of hori- 25 
zontal lines, wherein said three-dimensional filtering 
means comprises means for generating at least one high 
frequency component image emphasizing horizontal 
edges in said frames of said motion picture, and wherein 
said quantization means comprises means for approxi- 30 
mating the pixels of said high frequency component 
image emphasizing horizontal edges to a precision 
greater than said statistically optimum precision deter- 
mined by the statistical distribution of said pixels in said 
high-frequency component image emphasizing horizon- 35 
tal edges. 

4. The motion picture compression apparatus of claim 
1 wherein each said frame of said motion picture com- 
prises a luminance frame and first and second chromi- 
nance frames, wherein said three dimensional filtering 40 
means further comprises first and second three-dimen- 
sional chrominance filtering means, said first three-di- 
mensional chrominance filtering means comprising 
means for filtering N r of said first chrominance frames 

to generate a first set of three-dimensional filtered chro- 45 
minance component images, said second three-dimen- 
sional chrominance filtering means comprising means 
for Filtering the N, said second chrominance frames 
corresponding to N/ of said first chrominance frames 
filtered by said first three-dimensional chrominance 50 
filtering means to generate a second set of three-dimen- 
sional filtered chrominance component images, and 
wherein said quantization means further comprises vec- 
tor quantization means for approximating correspond- 
ing pixels in said first and second three-dimensional 55 
filtered chrominance component images by vectors of a 
predetermined precision. 

5. The motion picture compression apparatus of claim 
1 wherein said receiving means comprises: 

two-dimensional filtering means for filtering each of 60 
said received frames to generate a two-dimensional 
low-frequency component image and one or more 
two-dimensional high frequency component im- 
ages, each said component image comprising a 
plurality of pixel values, wherein 65 

said storing means comprises a plurality of frame 
buffers, each said frame buffer comprising means 
for storing said component images generated from 



one of said consecutively received frames, and 
wherein 

said three-dimensional filtering means comprises a 
one-dimensional filtering means for combining 
corresponding pixels in said frame buffers. 

6. The motion picture compression apparatus of claim 
5 wherein said two-dimensional filtering means com- 
prises a one-dimensional conjugate mirror filter bank. 

7. An apparatus for decompressing a compressed 
motion picture, said compressed motion picture com- 
prising a sequence of approximations for the pixels of 
one or more three-dimensional images, said decompres- 
sion apparatus comprising: 

means for receiving a signal specifying said pixel 
approximations; 

means for generating one or more three-dimensional 
component images from said received pixel ap- 
proximations including means for generating blank 
pixel values for any said pixel value not received; 

inverse three-dimensional filtering means for combin- 
ing said three-dimensional component images to 
generate a plurality of two-dimensional luminance 
picture frames; 

means for generating output signals specifying said 
generated luminance picture frames; 

8. The motion picture decompression apparatus of 
claim 7 wherein said pixel approximations comprise a 
plurality of two dimensional vector values, the compo- 
nents of each said vector value specifying first and sec- 
ond chrominance pixel values, wherein said means for 
generating three-dimensional component images com- 
prises means for assembling said first and second chro- 
minance pixel values into first and second sets of chro- 
minance component images, and wherein said inverse 
three-dimensional filtering means further comprises 
means for combining said first and second sets of chro- 
minance component images into first and second two- 
dimensional chrominance picture frames, and wherein 
said output signal generating means further comprises 
means for outputting said chrominance picture frames. 

9. A method for compressing a motion picture to a 
given compression ratio, said motion picture compris- 
ing a sequence of frames, each said frame comprising a 
two-dimensional image representing said motion pic- 
ture at one instance in time, said apparatus comprising: 

receiving signals specifying said frames; 

storing a predetermined number N* of consecutively 
received said frames; 

filtering said stored frames to generate a plurality of 
three-dimensional component images, said plural- 
ity of three-dimensional component images com- 
prising a low-frequency component image and one 
or more high-frequency component images, each 
said component image comprising a plurality of 
pixel values; and 

approximating each pixel values in at least one of said 
component images by an approximation compris- 
ing one of a predetermined set of integer values, the 
precision of said integers being related to the statis- 
tical distribution of the pixel values in said compo- 
nent image and said compression ratio; and 

generating output signals specifying said pixel ap- 
proximations. 

10. The method of claim 9 wherein said approximat- 
ing step comprises approximating the pixels of said 
low-frequency image to a precision greater than the 
statistically optimum precision determined by said sta- 
tistical distributions of said pixels in said component 
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imaees and said compression ratio: and approximating filtering each of said received frames to generate a 

saidpixels of at least one of said high frequency compo- two-dimensional low-frequency component image 

nent imaees to a precision less than ihe statistically and one or more two-dimensional high frequency 

optimum "precision determined bv the statistical distri- component images, each said component image 

butions of said pixels in said component images and said 5 comprising a plurality of pixel values wherein 

compression ratio. said storing step comprises storing said two-d.men- 

11 The method claim 9 wherein said frames of said «°nal component 'mages ma plurality of frame 

motion picture comprise interlaced frames, every other buffers, each said frame buffer ~mpna„g means 

said frame being offset by one half the horizontal line ^STtS^'SSS^ 

spacing of said frames, wherein said receiving step com- lu onc °! u 3 

prises combining each pair of consecutively received J combining correspond- 

frames to form a single frame having twice the number ^ ^ frame 

of horizontal lines, wherein said three-dimensional fil- H mcthod rf cW||| n wherejn said two ^ imen - 

tering step comprises generating at least one high fre- ^ siona] f|J| comprises filtering said pixels with a 

quency component image emphasizing horizontal one ^ imensional conjugate mirror filter bank, 

edges, and wherein said approximating step comprises J5 A rccording of a motion pict ure comprising a 

approximating the pixels of said high frequency compo- plurality of ordered frames, said recording comprising 

nent image emphasizing horizontal edges to a precision meam for storing a p ] ur ality of digital values, said digi- 

greater than said statistically optimum precision deter- 2Q tal values bcing divided into groups, each said group 

mined by the statistical distribution of said pixels in said comprising approximations for the pixels of one or more 

high-frequency component image emphasizing horizon- component images, said component images specifying a 

tal edges. three-dimensional image, each said three dimensional 

12. The method of claim 9 wherein each said frame of image comprising information specifying two or more 
said motion picture comprises a luminance frame and 2 5 frames of said motion picture, said component images 
firsi and second chrominance frames, wherein said fil- comprising a low-frequency component image and one 
tering step further comprises filtering N; of said first or more high-frequency component images, 
chrominance frames to generate a first set of three-di- 16. The recording of claim 15 wherein each said 
mensiona! filtered chrominance component images, group further comprises digital values specifying two- 
filtering the N/ said second chrominance frames corre- 30 dimensional chrominance vectors, said chrominance 
sponding to said N x said first chrominance frames to vectors specifying first and second three-dimensional 
generate a second set of three-dimensional filtered chro chrominance component images, said first and second 
minance component images, and wherein said approxi- chrominance component sub-images specifying a first 
mating step approximating corresponding pixels in said and second three-dimensional chrominance images, 
first and second three-dimensional filtered chrominance 35 17. The recording of claim 15 wherein each said 
component images by vectors of a predetermined preci- group comprises a high frequency component image 
sion. emphasizing horizontal image features in the frames of 

13. The method claim 9 wherein said receivinc step said motion picture. 

* • * * * 
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Title page: 

Item, [56] Attorney, Agent, or Firm - delete "Bartles" and insert therefor -Bartels-. 
Item, [73] Assignee - delete "Combridge" and insert therefor -Cambridge-. 
Column 2, line 52, after "amount" insert -equal-. 

Column 12, lines 19 and 20, delete "maybe stored back in the time component 
images,". 

Column 17, line 28, delete "As" and insert therefor -An-. 
[57] Abstract - delete "derrived" and insert therefor -derived-. 
Column 2, line 55, after "as" insert -a-. 

Column 10, line 57, delete "Analyzer" and insert therefor -analyzer-. 
Column 18, line 27, delete "image" and insert therefor -images-. 
Column 18, line 67, delete "values" and insert therefor -value-. 
Column 20, line 25, delete ";" and insert therefor 
Column 20, line 56, delete "values" and insert therfor -value-. 
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